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ABSTRACT 


In the field of human research, particularly in operational environments, data 
collection techniques are difficult. Researchers often focus their efforts on the data 
analysis and overlook the shortcomings of their data collection and storage 
methodologies. In order to demonstrate effective data collection and storage 
methodology in a representative human research process, the process used by human 
fatigue and performance researchers at the Human Systems Integration Lab at Naval 
Postgraduate School (NPS) served as a Proof of Concept. Most recent studies conducted 
at NPS provided a model of the current process. The Knowledge Value Added (KVA) 
methodology was used as a tool of comparison of the current process to the reengineered 
process. Information technologies including wireless physiological monitoring devices, 
web-based and mobile device data collection methods, and integrated data storage 
techniques were incorporated in the reengineering effort. The data storage process 
included the design of a standard relational database format allowing research teams to 
easily access their data. This repository also enables data to be archived for future use 
(e.g., meta-analyses). To demonstrate the reengineered process in an operational 
environment, a field fatigue study was conducted at the Naval Officer Indoctrination 
School (OIS) in Newport, Rhode Island. 
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1. INTRODUCTION 


A. OVERVIEW 

Data collection in the field of human research is intrinsically difficult. 
Monitoring physical and psychological behaviors through various data collection media 
and integrating the collected data points comprehensively for effective and timely 
analysis is a challenge in a carefully controlled laboratory environment. However, when 
the collection takes place in operational environments, the complexity intensifies. Often, 
the research team becomes focused on data analysis and they overlook major 
shortcomings in their collection and storage methodologies. Collecting human 
performance and fatigue data are one of the most complex research areas in the field of 
human research. In order to demonstrate effective data collection and storage 
methodology in a representative human research process, the process used by fatigue 
researchers at the Human Systems Integration Lab at Naval Postgraduate School (NPS), 
Monterey, California, served as a Proof of Concept. Various information technologies 
were considered in efforts to streamline the data collection and storage processes to 
improve the productivity of human research conducted under operational conditions. 

This process used in recent studies conducted by faculty and students at NPS of 
collecting and analyzing human performance and fatigue data were evaluated and 
reengineered. Web-based data collection methods and mobile communications devices 
were incorporated in the redesigned data collection sub-process enabling additional data 
points to be collected in a more efficient and timely manner with greater accuracy. The 
data storage process was redesigned as well by establishing a standard relational database 
format allowing research teams to easily access their data. This repository also enables 
data to be archived for future use (e.g., meta-analyses). A study was conducted at the 
Naval Officer Indoctrination School (OIS) in Newport, Rhode Island to demonstrate the 
newly optimized process. 

B. BACKGROUND 

To improve the real life human performance, human research must be conducted 
in operational settings. Because of the difficult challenges environmental conditions 

place on such research, it is even more important for researchers to collect data efficiently 
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and as unobtrusively as possible. There is often only one chance for (hta collection. 
Since so much is at stake under these conditions, it is essential that perhaps the only 
chance to capture human statistics under such circumstances not be lost due to poor data 
collection methodology. Of all of the areas of human research, fatigue as it relates to 
human performance is particularly complex. Additionally, it has become of increasing 
concern for the military as the force has been tasked to accomplish more with less people. 

Fatigue management has become a significant issue in effective warfighting. The 
news media, embedded with the forces on the ground and at sea during the war in Iraq, 
continuously reported the sleep deprived state of the men and women serving in the 
armed forces and the potential dangers fatigue had on their vigilance while operating 
technical systems. The awareness of the consequences of fatigue continues to grow. As 
this occurs, a desire for innovative solutions to mitigate the given combat circumstances 
and achieve maximum human performance is at a record high. 

At NFS, the current process of collecting fatigue data in an operational 
environment, analyzing the data, and publishing the results takes about a year to 
complete. The researchers for this process are thesis students who require significant 
training in the science of conducting human research as well as how to use the collection 
equipment and analysis tools available to them at NFS. In order to arm future sleep 
research teams with the ability to identify sponsoring organization’s fatigue issues in a 
timely manner, the current fatigue study process has to change. Using this current 
process, identifying potential danger areas is the extent of the analysis. There is little 
time available to propose innovative solutions for the sponsoring organization. The new 
process must reduce the time required to conduct the study and provide for collection of 
enough quality data to enable the team to draw conclusions and recommend solutions or 
at the least, identify areas to focus further study if required. Additionally, a capability to 
enable meta-analysis of many studies conducted by NFS students and faculty would 
clearly provide a powerful tool not currently available as data are stored in disparate 
structures using many unique systems. 
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C. RESEARCH QUESTIONS 

• How much can the productivity of the current process of data collection of 
human performance and fatigue data during studies conducted at the NPS 
be improved through redesign using information technology? 

• How will web-based technology facilitate data collection in the process 
redesign? 

• How will modern database technology enable efficiently structured data 
storage in the process redesign? 

• How will current mobile personal computing device (PDA) technology 
facilitate data collection and integration in the process redesign? 

• How will performance testing using a PDA aid the process 
redesign? 

• How will development of a PDA based tracking log give support 
to process redesign? 

D. SCOPE OF THESIS 

In efforts to improve data collection and storage processes in the field of human 
research, the current fatigue study process at NPS was examined as the Proof of Concept. 
The process was analyzed, evaluated, and reengineered using information technology to 
provide an optimal solution. The most significant areas of improvement included the 
innovative data collection and data storage methods. The optimized model was tested as 
data were collected, analyzed, and stored. After necessary revisions to the model, an 
operational study was conducted at the Navy OIS in Newport, Rhode Island to validate 
the reengineered process. The scope is limited to the collection and storage of data. 
Research efforts toward interpretation of the data collected during this study were 
focused to validate the reengineered process rather than identify issues of fatigue of the 
study participants. 

E. RESEARCH METHODOLOGY 

To determine the best approach to reengineer the process, the current methods of 
collection and analysis were studied and the Knowledge Value Added (KVA) 
methodology was used. KVA provides a way to capture knowledge within a set of 
processes and measure its associated value. The KVA methodology provided the ability 
to evaluate the current process and establish a baseline for comparison to the 
reengineered process. The redesigned process was developed using a three tiered 


3 



architecture approach and was evaluated using the KVA method. Finally, this process 
was tested in the Human Systems Integration Lab at NFS, fine-tuned, and was ultimately 
field tested at the Navy OIS in Newport, Rhode Island. 

F. THESIS ORGANIZATION 

Chapter II follows with a Literature Review including an overview of human 
performance and effects of fatigue as well as a description of the software tools used 
during fatigue studies for data collection and analysis. The current process of conducting 
these studies is outlined in Chapter III. An explanation of the process model and an 
evaluation using KVA analysis is provided. Chapter IV summarizes the technical 
approach used in reengineering the current process to include the parameters of design 
and the data collection and storage methods considered. Chapter V provides the model 
for the reengineered process and an evaluation of it using the KVA methodology. The 
validation of the new process is explained in Chapter VI with a description of the study 
conducted with the OIS participants in Newport, Hiode Island. Finally, Chapter VII 
concludes with recommendations for future work. 
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n. LITERATURE REVIEW 


A. OVERVIEW 

This literature review covers several diverse areas. To assist in evaluation of the 
data collection and storage processes of the Proof of Concept, this review is meant to 
provide the reader with an overall understanding of the field of sleep, fatigue and human 
performance and relevant research techniques. In order to adequately assess the results of 
the reengineered process, an introduction to the Knowledge Value Added methodology is 
also presented. Though the focus of this review is on the data collection techniques 
specific to the fatigue research process, the methods of data collection presented are 
intended to serve as an example of the numerous means of data collection used in human 
research. 

The effects of fatigue and sleep deprivation on human performance have been of 
keen interest to those in the field of sleep research and those in the business of optimizing 
human performance effectiveness. This review describes human physiology and the role 
sleep plays in biological systems. The impact of sleep deprivation on human physiology 
and performance is introduced as well as reports of findings in recent dose-response 
studies. Additionally, current methods of data collection and analysis used to document 
fatigue as it relates to human performance are described. 

As the military continues to require warfighters to operate under conditions of 
chronic sleep deprivation, the risks have become clear. Recent studies conducted by 
students and faculty at NFS highlight the military’s continued interest in documenting the 
effects of fatigue on human performance in operational environments. These efforts 
provide the necessary evidence to implement changes enabling mitigation of risks 
imposed by lack of sleep. Following these descriptions, the current tools used during 
sleep studies by NFS research teams are introduced. This section discusses data 
collection tools as well as the software required to download and analyze fatigue data. 
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In the final section of this chapter, an overview of the Knowledge Value Added 
approach to assessing the value of knowledge in a given process will be introduced. This 
methodology provides the means to compare the current process used by these recent 
NPS studies to the reengineered process proposed from this research. 

B. HUMAN PERFORMANCE AND FATIGUE 

1. Human Physiology and the Role of Sleep 

Human beings, evolving over time, have developed an internal biological clock 
that serves to regulate sleep and wake periods. These endogenous clocks have been 
described to control a number of biochemical, physiological, and behavioral phenomena 
in humans. Operating on a cyclic pattern, these clocks repeat approximately once per day 
and are called circadian rhythms. Franz Halberg of the University of Minnesota first 
used the word circadian as it comes from the Latin ‘circa’ (about) and ‘dies’ (day). In the 
late 1960s, scientists, including Jurgen Acshoff, at the Max Planck Institute in Germany 
isolated individuals from any external environmental cues and allowed the human 
participants to set their own schedules. After days of isolation, participants adjusted to a 
regular cycle of sleeping and waking. These scientists also discovered that the internal 
clock cycle repeated approximately every 25 hours (Dement, 1999). 

Because the 25 hour cycle time varies slightly from the Earth’s rotation cycle, it is 
necessary for humans to receive environmental time cues called ‘zeitgebers’ which can 
help to reset the biological clock. The location of the internal clock appears to be in a 
small area of the hypothalamus known as the suprachiasmatic nucleus (SCN). It is 
sensitive to the inputs from the optic nerve. These inputs act as a controlling mechanism 
making necessary adjustments to the internal clock. Because of this, light is the most 
powerful zeitgeber for humans. Charles A. Czeisler performed several studies to 
demonstrate light’s influence on circadian rhythms. His results conclude that even dim 
light affects this circadian clock (Czeisler, 1990). Figure 1 shows the pattern for typical 
alertness over the course of a day. The effects of accumulated sleep dept as a result of 
extended wakefulness counter the effects of the biological clock. Drs. Charles Edgar and 
William Dement defined an opponent-process model which suggests that the alerting 
mechanism of the biological clock as well as a sleep drive that increases during waking 
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hours as sleep debt accumulates are the two processes that regulates sleep in humans. 
The alerting mechanisms occur when a person awakens in the morning and in the late 
afternoon (Edgar et. al, 1993). 
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Figure 1. Alertness Over 24-Hour Period (From: Maas, 1998) 

2. Performance Degradation Due to Fatigue 

As neurological studies continue to examine what the brain and other body 
functions do during sleep, the exact activities remain largely unknown. Progress has 
been made in assessing the effects that sleep deprivation has on performance. 
Researchers agree undisputedly that total sleep restriction results in huge performance 
decrements. The extent of the performance degradation with partial sleep deprivation is 
currently the focus of much research. When comparing the effects of chronic partial 
sleep deprivation to the effects of complete sleep restriction, the resulting performance 
decrement is surprisingly similar. In a recent dose-response study conducted at Walter 
Reed Army Institute of Research, individuals were allowed to sleep for a restricted 
amount of time over a seven day period. Sixty-six people spent either 3, 5, 7, or 9 hours 
in bed during this period (Belenky, et. al, 2003). Over the course of the study, the 
participants conducted psychomotor vigilance tests to measure their simple reaction time. 
The participants pushed a thumb operated handheld device as a reaction to a visual 
stimulus. The results of their psychomotor vigilance testing (PVT) can be seen in Figure 
2. As expected, PVT speeds decrease over time with the most dramatic changes seen in 
the group who spent three hours a night in bed. All of the participants were given a three 
day recovery period where there were no restrictions on sleep. After three days, none of 
the sleep deprived groups returned to their baseline performance speed. The conclusions 
drawn by the researchers suggest that the human body, in a state of chronic deprivation. 


7 



adapts by stabilizing the individual’s performance to a level below that of the initial 
baseline. This stabilization, however, prevents the rapid recovery to baseline 
performance. This finding is in contrast to the rapid recovery in performance seen in 
individuals whose sleep has been completely restricted. 



Figure 2. Mean Psychomotor Vigilance Task Speed (and Standard Error) 
Across Days as a Function of Time (From: Belenky, et al, 2003) 

A similar study conducted at the University of Pennsylvania restricted participants 
to either 4, 6, or 8 hours of sleep per night for a period of 14 days. Another group was 
kept awake for 88 hours. All participants were given a performance assessment battery. 
This battery included a psychomotor vigilance test similar to the one used in the Walter 
Reed study to measure behavioral alertness as well as a computerized digit symbol 
substitution test to measure working memory. Researchers found that subjects whose 
sleep period was 4 to 6 hours a night had significant deficits in cognitive performance, 
although the subjects reported being only slightly sleepy. Those participants who were 
completely deprived of sleep had cognitive deficiencies similar to the reduced sleep 
group; however, they reported being very tired. The results indicate that although partial 
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sleep deprivation significantly decreases performance, individuals are unaware of this 
decrease in efficiency (Van Dongen, et. al, 2003). These study results present a clearer 
picture of possible dangers to members of the military as well as medical professionals 
and shift workers. In occupations where 6 hours of sleep a night or less is the norm, 
individuals may be operating at decreased performance fevels without even knowing it. 
The potential risks associated with the errors made when performance degrades are 
alarming. 

3. Data Collection Techniques (Physiology and Performance) 

Human research includes the collection of both physiological and psychological 
data points. The field of biometrics has expanded so much recently that individuals 
conducting human research or simply gathering human physiological data have many 
wireless devices available today for data collection. From portable heart monitors to 
caloric intake measurement devices, innovative measurement techniques are allowing 
research to be conducted in much more realistic environments. The explosion of wireless 
technology in computing allows PDAs and other handheld devices to enable the 
collection of an endless assortment of data including but not limited to survey data and 
human performance data with minimal disruption to the individual participating in the 
research. As examples of some of the more intricate methods of data collection involving 
humans, fatigue and human performance measurement techniques provide challenges to 
field researchers. 

Assessments of fatigue and human performance in particular have typically been 
accomplished within the laboratory setting using a variety of standard measurement tools. 
Sleep researchers frequently use electrophysiology to assess sleep. Human adults 
experience five stages of sleep during a typical night’s sleep: stages one through four and 
REM (Rapid Eye Movement) sleep. Stages one through four are referred to as Non-REM 
(NREM) sleep. Each sleep stage has its own identifying marks as measured by 
electroencephalographs (EEGs), electromyograms (EMGs), and electrooculograms 
(EOGs). EEGs measure brain waves via electrodes pasted to the scalp. EMGs measure 
muscle activity and EOGs measure eye movements also using contact electrodes. The 
collective group of EEG, EMG and EOG measurements taken when studying sleep stages 
is referred to as Polysomnography (PSG). The physiologic sensor leads are placed on the 
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participant in order to record any number of the following activities: brain electrical 
activity, eye and jaw muscle movement, leg muscle movement, airflow, respiratory 
effort, and oxygen saturation. Although these measurement techniques have proven to be 
very effective, they are highly sophisticated, very expensive, and extremely obtrusive to 
the individual. 

When collecting data on participants in their work environments as the field 
studies conducted by NPS research teams do, less obtrusive means of measuring sleep are 
required. The use of actigraphy to measure movement has become the most effective 
means for researchers to capture sleep/wake periods of study participants in field studies. 
Many types of small wireless monitoring devices are available on the market today that 
provide the capability to measure a number of physiological activities. Wrist activity 
monitors or WAMs are the size of a wrist watch and contain an accelerometer that 
measures movement. The associated software of the WAMs use algorithms to calculate 
sleep and wake periods based on measured activity levels. WAMs, however, do not 
allow the researcher to measure brain activity. Therefore, in operational studies where 
WAMs are used, researchers cannot measure the stages of sleep participants reach but 
only can determine when the participants slept and when they were awake. Sleep 
efficiency can be calculated from the WAM data as a ratio of sleep epochs to actual sleep 
period. Sleep latency can also be calculated from the time an individual goes down for 
sleep until the time he is actually asleep. The various WAMs on the market today use 
different methods to record movement and offer different software tools to provide 
unique analytical techniques. 

Collecting additional variables such as melatonin levels or ambient light exposure 
are useful techniques used in predicting circadian rhythms. Researchers measure 
hormone levels such as melatonin by collecting saliva or blood samples which have a 
distinctive circadian pattern. Other measurements such as light exposure, body 
temperature or subjective activity logs are useful for assisting researchers in 
understanding circadian patterns as well as interpreting individual behavior and 
predicting effectiveness. 
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Various techniques have been developed to measure performance or model 
predicted effectiveness. Cognitive testing is one approach to measuring effectiveness. 
Psychomotor vigilance tests including reaction time tests, running memory tests, math 
processing tests, and others are used by researchers to measure performance. One of the 
tools created specifically for field environments is the ARES (ANAM^'^ Readiness 
Evaluation System). Included in this tool is the Stanford Subjective Sleepiness Scale that 
identifies the particular sleepiness state felt by the participants a given time. One tool 
used for predicting cognitive effectiveness is the Eatigue Avoidance Scheduling Tool 
(EAST) built on principles of the Sleep, Activity, Eatigue, and Task Effectiveness 
(SAETE) Model. Analysis of data collected on participants engaged in the performance 
of cognitive tasks during several sleep deprivation studies conducted by the Army, Air 
Eorce and Canadian researchers is the basis for the expected level of performance 
effectiveness predicted by the EAST. Though designed as a scheduling tool, the EAST 
can serve as a model to determine effectiveness levels based on collected WAM 
accelerometer data. 

C. RECENT STUDIES CONDUCTED AT THE NAVAU POSTGRADUATE 

SCHOOU 

As recent studies continue to show the strong correlation between sleep 
deprivation and the degradation of human performance, the United States military has 
developed a keen interest in studying effects of fatigue on service members. Students 
and faculty at NPS have recently conducted studies to assess these effects in various 
operational environments in the Navy. 

ET John Nguyen collected data on the aircraft carrier the USS JOHN C. 
STENNIS during a shift from day operations to night operations in support of 
OPERATION ENDURING EREEDOM. He used Actigraphs to collect actigraphy and 
sleep data and required the participants to keep a daily journal of their sleep activity. He 
also collected oral temperatures of the participants every three waking hours and required 
each participant to fill out several surveys to support his research. His thesis revealed the 
difficulty experienced by sailors when reversing sleep schedules from day to night 
operations. Additionally, ET Nguyen’s data exposed significant differences between 
those sailors working topside and those working below decks. As a result of his findings. 
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interest has been generated to more closely understand the effects that ambient light and 
other zeitgebers have on sailors in operational situations. While it is understood that 
mission demands may not allow optimal sleep schedules and sterile environments to be 
created and applied on deployed carriers, this study concluded that investigations into 
optimizing schedules and environments within the constraints of a given deployment may 
be worth the effort. Potentially maximizing human performance effectiveness as it 
relates to sleep deprivation in the arduous environments they face at war is the possibility 
highlighted in LT Nguyen’s research. 

During another study conducted at the Great Lakes Naval Recruiting Station, LT 
Brian Baldus examined the effects of changing bedtime for Navy recruits from 2100 to 
2200. Using the WAMs and various subjective tests, he concluded that the change in 
time was significant between the two sleep condiions (Baldus, 2002). He concluded that 
recruits who were able to go to bed later were able to get to sleep faster and therefore 
sleep longer. This increase ultimately improved their performance effectiveness. 
Because the average ages of the study population are adolescents and young adults, the 
later bedtime aligned more appropriately with the known differences in adolescent/young 
adult circadian rhythms. Other training institutions throughout the military have shown 
interest in these results, prompting the desire for additional studies to find optimal sleep 
plans for unique circumstances in training as well as in operational environments. 

Two other recent NPS theses examine sleep in operational settings (Blassingame, 
2001 and Gamboa, 2002). In an analysis of archival submarine survey data collected by 
the Naval Submarine Medical Research Laboratory (NSMRL), LT Donnie Gamboa and 
LT Simonia Blassingame revealed that sailors have shown characteristics of sleep 
deprivation that they claim inhibit their ability to stand watch at an optimum level. His 
research focused on answering questions related to environmental constraints and time in 
service as factors related to sleep and fatigue. LT Gamboa also presented a useful 
discussion regarding the importance of sleep management in optimizing operational 
performance. His conclusions emphasize the requirement to manage sleep as a resource 
for warfighters. He says the potential for mismanaged sleep could most certainly result 
in missions being compromised (Gamboa, 2002). 
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D. MEASUREMENT TOOLS AND SOFTWARE USED IN NFS STUDIES 

The specific measurement techniques and tools used in recent NFS fatigue and 
performance studies are described in the following paragraphs. The particular WAM 
device used is called an Actigraph. It was developed by the Precision Control Design, 
Inc. (PDC). The sleep watches, along with accompanying software to initialize and 
download the data, are marketed by Ambulatory Monitoring, Inc. (AMI). Dr. Timothy 
Elsmore from Activity Research Services, Inc. has authored the Action-W software 
(marketed by AMI) which is a Windows-based program used for analysis of Actigraph 
data. The Fatigue Avoidance Scheduling Tool (FAST), developed by NTI, Inc., was used 
to model the performance effectiveness of the participants based on the WAM data. 
Finally, ARES software used on the handheld PDA devices, developed by Dr. Timothy 
Elsmore, provides researchers with a performance data point based on measurements 
taken from tasks conducted by participants throughout the study. 

I. Actigraphs and ACT Millennium Software 

There are several types of commercial WAMs that monitor activity levels 
available on the market. NPS research teams have used the Octagonal Sleepwatch® 
(Sleep-O Model) Actigraph and associated Altigraph reader interface made by PDC (see 
Figure 3). The Actigraph contains an accelerometer that can be set to record activity in 
epochs varying from one second up to one minute. All Actigraphs used in NPS studies 
were set to collect data in one-minute epoch lengths. The ACT Millennium software is 
the Windows based computer program used to initialize the Actigraphs and download the 
data from them. The watch is placed on an Actigraph reader (shown in Figure 3) that 
allows communication between the ACT Millennium software and the Actigraph. Focal 
time, type of Actigraph, sampling modes (e.g.. Zero Crossing mode. Time Above 
Threshold mode, and Proportional Integrating Measure mode), epoch length, Actigraph 
identification number, when to start recording data, estimated runtime, and a header 
preview are all reviewed and set during initialization. ACT Millennium also has a 
diagnostics function that allows the research team to check the settings of the Actigraph 
reader and to ensure the accurate transmission of data. 


13 




Figure 3. Octagonal Sleepwatch® and Reader Interface 

Figure 4 shows the ACT Millennium start up screen. ACT Millennium allows the 
Actigraph data to be saved with a .DAT file extension which can be read by ACT 
Millennium software or with the .AMI file extension that can be read by another AMI 
software program called Action W. 



Figure 4. ACT Millennium Start Up Screen 
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2. Initial Processing of Actigraph Data: Action W 

Ambulatory Monitoring, Inc. also markets the Action W software. Action W allows 
visualization of the activity levels for the entire data collection period. A screen capture 
of an individual’s sleep data are displayed in Figure 5. Action W scores the sleep and 
wake periods and offers manual trimming of the data if the researcher deems it necessary. 
The red underline indicates sleep and the green shaded portion represents down periods. 
Basic statistics can be run using Action W such as total sleep minutes, wake minutes and 
percentages of time spent sleeping. The Cole-Kripke algorithm is used when scoring the 
sleep and wake periods. This method, developed in 1988 and revised in 1992, uses a 
weighted average of the previous four minutes, the current minute, and the next two 
minutes in order to determine whether the current minute should be labeled as ‘sleep’ or 
‘wake.’ The researcher can code the data to override the automatic scoring of the 
accelerometer to allow for the correction of the more obvious discrepancies that may 
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Figure 5 Action W Linear Actigraph View of Sample Individual’s Data 
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result from using the weighted average. Action W allows the researcher to save a file 
with the file extension .AMI for use again in Action W or as an epoch by epoch (.EBE) 
format for use with EAST. 

3. Fatigue Avoidance Scheduling Tool (FAST) 

FAST is a computer program developed by NTI, Inc. designed to allow 
schedulers to estimate the effects of various schedules on human performance. It also 
allows WAM data to be imported into a FAST schedule. Using the Actigraph watch, for 
example, researchers at NFS have been able to “clean” the collected data using the 
Action W tool and then import that data to the FAST tool. This allows the researcher to 
see the modeled performance effectiveness levels of the participant throughout the course 
of the study. FAST gives estimates of an individual’s predicted effectiveness based on 
the SAFTE model described earlier. It uses the SAFTE algorithm created by Dr. Steve 
Hursh (SAIC) in conjunction with the Walter Reed Army Institute of Research 
(WRAIR). The schematic of the SAFTE Model is diown in Figure 6. It integrates 
quantitative information about circadian rhythms in metabolic rate, cognitive 
performance recovery rates associated with sleep, and cognitive performance decay rates 
associated with wakefulness, and cognitive performance effects associated with sleep 
inertia (Hursh et. al, 2003). The software also allows the research team to pre-condition 
the starting condition for an individual to adjust for differences in sleep prior to the actual 
data collection period. This allows the level of the individual’s sleep reservoir to be set 
based on more realistic conditions. FAST then analyzes each minute of data imported 
and graphically represents the individual’s sleep efficiency. 
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Schematic ol'SAFTE Model 

Sleep, Activity, iatiyue and iask Effeetiveness Model 



Figure 6. Schematic of the SAFTE Model (From: Hursh et. al, 2003) 

A screen capture with explanation is shown in Figure 7. Sleep intervals and work 
periods are depicted on the horizontal axis. The vertical axis represents the effectiveness, 
and the right hand side shows a blood alcohol equivalent measure. The main display is 
shown in colors that represent degrees of effectiveness. The upper section is green 
indicating excellent effectiveness, followed by yellow indicating reduced effectiveness. 
The bottom portion is pink/red and indicates less than 65% effectiveness. A dotted line is 
shown at the 78% effectiveness level; a level chosen by the U.S. Air Force as the “floor” 
for predicted effectiveness for their pilots. Additional tools allow the researcher to view 
and manipulate the data in many different ways, such as minute-by-minute effectiveness 
or by altering start times and durations of wake periods. If the Action W data are not 
accurately trimmed to identify true sleep/wake periods, the FAST data will be skewed. 
Requiring the individual participants to maintain a log of their activity during the data 
collection period allows researchers to trim the collected Actigraph data effieiently. The 
quality of these logs may determine the accuracy of the FAST modeled outputs of 
performance effectiveness. 
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Figure 7 FAST Display 
4. ARES (ANAM™ Readiness Evaluation System) 

ARES is automated neuropsychological testing software for operation on 
handheld PDAs that use the Palm Operating System (OS) Software Version 3.5 or later 
that was developed by Dr. Timothy Elsmore. The system is designed to run test batteries 
that are intended to meet specific clinical, operational, or research goals. These tests 
include Sleep Scale, Mood Scale 2, Simple Reaction Time, Code Substitution (Learning), 
Math Processing, Matching to Sample, Logical Relations, Code Sub (Delayed Recall), 
and Memory Search. The ARES will operate on a wide range of PDAs including those 
with black and white or color displays. The start-up screen is shown in Eigure 8. 
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Figure 8. PDA Device (Palm mlOO) with ARES Software 

The ARES allows multiple users on a single PDA, and includes a variety of 
standard test batteries and a means to install custom batteries. The data saved on the 
PDA can be downloaded to a desktop data management program, DataMan, using a 
direct serial link, USB connection or wireless modems. The data can be downloaded 
using the HotSync Manager associated with most PDA software or by importing directly 
from the handheld device to the DataMan software. When using HotSync, ARES data 
are stored in a Palm Database (.PDB) file format which in turn can be accessed using the 
DataMan software. The DataMan program has limited data analysis functions and allows 
the data to be copied easily into Microsoft Excel or Microsoft Access tables for further 
analysis. The NPS Human Systems Integration Eab (HSIE) currently uses the 
Handspring Treo 90 as the PDA device of choice based on cost and functionality. 

E. KNOWLEDGE VALUE ADDED METHODOLOGY 

To evaluate the reengineered process as compared with the current process, the 
Knowledge Value Added (KVA) methodology was used. Developed for use in an 
economic setting, this KVA methodology is often used to determine the value 
information technology adds to a given process. Other methods of assessing value 
include Cost-Based Approaches, Market Approaches, Income Approaches, and Real 
Options Approaches. Because of the nature of this comparative analysis, it was 

necessary to choose a methodology that allowed a valuation for knowledge. Because 
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revenue is not generated in these processes, a surrogate measurement was required. KVA 
provided the means to measure the inputs to the current process while capturing the given 
outputs and compare this to those of the reengineered process. 

KVA is based on the principle of replication that states “given that we have the 
knowledge necessary to produce the change, then we have the amount of change 
introduced by the knowledge” (Housel and Bell, 2001). If a process input is the same as 
its output, there was no change, therefore, no value has been added. Representing a 
process as a function P(X) where P(X) = Y, it follows that if X = Y, there was no change 
and no value added by the process. Value is assumed to be proportionate to change. 
Change can be measured by the amount of knowledge required to make the change. By 
definition, value is proportionate t) change which is proportionate to the knowledge 
required to make the change. To compare the current sleep study process to the 
reengineered process, it was necessary to measure the amount of knowledge required to 
produce the output of the core sub-processes and measure the cost of using this 
knowledge to find the Return on Knowledge (ROK) as the factor of comparison. ROK is 
the amount of knowledge divided by the cost. This ROK is a corporate performance ratio 
that gives order of magnitude comparisons for all of the processes. The amount of 
knowledge is determined by process analysis and interviews with subject matter experts 
to quantify the learning time (relative and actual) for each process. Process instructions 
or bits required to produce process outputs are other measurement techniques to obtain 
the amount of knowledge. The cost is calculated by multiplying the cost for the resource 
by the amount of time required to use the knowledge to produce an output. The resulting 
cost number is applied directly to the output for the process. 

The seven step method (Housel and Bell, 2001) to find the ROK for both the 
current and reengineered process was used to measure how much the productivity 
improved using information technology from the current or “AS-IS” process to the 
reengineered or “TO-BE” process. The first step is to identify the core process and sub¬ 
processes. In step 2, common units of measure for learning time are established. 
Calculation of the learning time to execute each of the sub-processes is step 3. In step 4, 
the sampling time period is determined to capture a representative sample of the core 

process’s final product or output. In step 5, the learning time for each sub-process is 
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multiplied by the number of executions of that sub-process during the sample period to 
get the numerator for the ROK. Similarly, in step 6 the denominator for the ROK is 
determined by calculating the costs for each sub-process. In the final step, the ROK is 
calculated and the results are interpreted. 
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m. CURRENT PROCESS EOR DATA COLLECTION AND 
ANALYSIS IN RECENT NPS EATIGUE STUDIES 

A. OVERVIEW 

Because the reengineered process for fatigue studies conducted by HSIL at NPS 
serves as the Proof of Concept for the data collection and storage methodology for human 
research in field environments, it was necessary to articulate the current or “AS-IS” 
process used in recent NPS fatigue studies. The Knowledge Value Added (KVA) 
approach to this process reengineering effort provides an objective method for measuring 
the amount of value a given component process adds as well as the cumulative value of 
the entire process. This enabled the comparative analysis of the current or “AS-IS” 
process to the process after reengineering efforts or the “TO-BE” process. To compare 
this value added, we had to calculate the Return on Knowledge (ROK) at the process and 
sub-process levels for both the “AS-IS” process and the reengineered or “TO-BE” 
process. 

In this chapter, the seven step KVA process was used to determine the “AS-IS” 
ROK. Because the “AS-IS” model is a reflection of the current process used by HSIE at 
NPS, summaries of the techniques used in two recent NPS studies are provided to lay the 
foundation for the “AS-IS” model development. A review of the seven step process is 
described and then applied to determine the ROK for the “AS-IS” process. Einally, an 
assessment of the KVA analysis results follows with the identification of the focus areas 
for the reengineering effort, revised data collection and storage methodologies. 

B. RECENT NPS FATIGUE STUDIES 

Two recent studies conducted by NPS thesis students included the U.S. Navy 
Recruit Training Command at Great Eakes, IE (NTCGE) Study (Baldus, 2002) and the 
USS JOHN C. STENNIS Study (Nguyen, 2002). While the objectives of these studies 
differed, the methods used for data collection were similar. In both studies, Actigraph 
watches were used to monitor participants’ activity levels while other data were collected 
on the participants. In the Great Eakes study, recruit test scores, demographic and other 
data were collected via various databases through local points of contact at Great Eakes, 
Naval Training Center (Baldus, 2002). Similarly, in the USS JOHN C. STENNIS study, 
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demographic data and data from a momingness/eveningness survey (Nguyen, 2002) were 
collected. To collect his information, LT Nguyen provided paper surveys for his 
participants to complete. In addition, the USS JOHN C. STENNIS study participants 
were instructed to maintain daily paper journals to record their sleep behavior as well as 
to record their oral temperature readings. In the Great Lakes study, no journals or logs 
were maintained by the participants. Because of the operational environments of these 
two studies, the researchers wanted to minimize the obtrusiveness of their data collection 
and as a result, made every effort to mitigate the burden on participants while optimizing 
the data collected. 

The data collection methodology for both of these studies required a tremendous 
amount of human intervention by the researcher. After the time consuming effort 
involved in transposing data from manual pen and paper entries from the USS JOHN C. 
STENNIS participants and the printed data sheets from the Great Lakes participants to 
electronic MS Excel spreadsheets, many hours were spent cleaning the data. Because the 
data from the USS JOHN C. STENNIS study were collected and recorded before the start 
of the Great Lakes study, the lack of confidence in participants’ ability to maintain 
accurate logs under operational constraints drove LT Baldus’ decision to exclude 
participants from recording daily activity. Additional variables were of interest to both 
researchers but the uncertainty of the collection methods prevented their inclusion in the 
studies. Eor example, both researchers had desires to collect performance data but had no 
feasible means to do so. While performance testing methods were available, the 
unfamiliar logistics involved in using a viable collection method was a major obstacle 
preventing the researchers from collecting performance data. 

C. KVA METHODOLOGY: THE SEVEN STEP APPROACH TO 

DETERMINING RETURN ON KNOWLEDGE 

In the audit of the “AS-IS” process, we evaluated the Return on Knowfedge 
(ROK) at the process and sub-process levels. To determine ROK, we measured learning 
time and cost. Learning time was used as a surrogate to measure knowledge embedded in 
the processes. Learning time is the amount of time it would take for someone to learn 
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how to do each activity in the process. The cost refers to how much money is required to 
use this knowledge each time the process or sub-processes are executed. After gathering 
the learning time and cost information, the Return on Knowledge (ROK) was calculated. 

The following seven-step methodology measures processes in terms of their 

ROK. 

STEP 1: Identify the core process and its component processes 

STEP 2: Establish common units to measure learning time 

STEP 3: Calculate learning time to execute each component process 

STEP 4: Designate a time period long enough to capture a representative 

sample of the organization’s final product/service outputs following common statistical 
sampling practices 

STEPS: Multiply the learning time for each component process by the 

number of times the component executes during the sample period 

STEP 6: Calculate the total cost to produce the output for each component 

STEP 7: Compute the ROK for each component process 

Since this method relies on the estimations of Subject Matter Experts (SMEs) for 
process learning time to gain a high-level ROK assessment of a given process, it is often 
useful to use another surrogate for knowledge such as process instructions to ensure 
validity of the analysis. In this Proof of Concept effort, we consulted with the SMEs and 
developed a process model using actual learning times for each sub-process. The 
granularity of this effort provided sufficient evidence to the stakeholders at NPS that 
further assessments were not required. The same level of granularity was applied to the 
reengineered “TO-BE” process allowing viable comparisons to be made. 

D. CURRENT PROCESS MODEL 

To clearly represent the process model for the past NPS fatigue research studies, 

we interviewed Dr. Nita Miller, head of the Human Systems Integration Eab (HSIE) at 

NPS, and several of her thesis students who have conducted data collection and analysis 

for fatigue studies. Additional interviews were conducted with other DoD SMEs who are 

involved in similar processes of collection and analysis. Observations of the NPS 
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researchers were also recorded during the latest studies to allow the creation of a 
representative model of the “AS-IS” process. Applying STEP 1 of the seven step KVA 
methodology, the core process of Conducting a Fatigue Research Study was identified. 
Three basic phases were determined as sub-processes of the model: 1) Collect Data, 2) 
Prepare Data for Analysis, and 3) Analyze and Publish Data. 

1. Collect Data Sub-Process 

In the Collect Data Sub-Process shown in Figure 9 below, the activities are 
depicted with circles while the outputs of those activities (or processes) are indicated by 
squares. The sub-process begins with a requirement for research resulting in a request for 
the HSIF to conduct a study. Included in this input request are the study parameters: 
purpose of the study, permission for study from customer requesting research, 
identification of the research team, and methods of data collection. Because the model 
for NPS research centers on students collecting and analyzing the data, training of the 
research team is essential for each study conducted. This Train Research Team Activity 
focuses on teaching research methods to the research team and includes instruction on 
study design, methods of data collection and analysis, and other unique considerations of 
conducting human research in operational environments. The output of the activity is a 
Trained Research Team. 





Completed 
Paper 
Surveys, 
Activity/ 
Temp/Sleep 
Rating Log 

Watches with 
stored collected 
data 

(thermometers) 


Figure 9. Collect Data Sub-Process 


The next phase of this sub-process is to define the conditions under which the data 
will be collected. Under the Preparing for Data Collection Activity, there are several 
tasks required including: 


26 









• Design the Study 

• Identify Funding Requirements 

• Create Demographic Survey 

• Type Lark and Owl Survey 

• Create Activity/Temp/Sleep Rating Log 

• Type Consent Forms 

• Produce Institutional Review Board Proposal 

The output documents from these tasks are the completed study design, approved 
Institutional Review Board Proposal, funding requirements, typed surveys, logs, and 
consent forms. 

Just before data collection actually starts, the required equipment which is to be 
used in the collection is prepared. The tasks for this Prepare Equipment Activity include 
initializing the Actigraphs, obtaining thermometers when participants’ temperatures will 
be recorded, and making copies of all surveys and forms for all participants. The output 
is all of the paperwork and equipment prepared for collection. Once on location, the 
research team then has to identify the study participants. Because of operational 
constraints, pure random sampling is almost never an option but the researchers try to 
select a sample of participants that will be representative of the entire group. Once 
selected, the participants fill out consent forms enabling the research team to conduct the 
study. 

The next activity. Distribute and Collect Surveys, Equipment, and Eogs, includes 
both issuing the equipment and the instructions required of the participants as well as the 
collection of the equipment upon completion of the data collection phase. The output of 
these tasks includes the surveys and logs completed by participants as well as the 
Actigraphs with recorded actigraphy in their memory, and the thermometers. Once all 
equipment and data are collected, the Collect Data Sub-Process is complete. 

2. Prepare Data for Analysis Sub-Process 

With the collected equipment, paper surveys, and logs, the research team returns 
to HSIE at NPS and begins the Prepare Data for Analysis Sub-Process represented in 
Eigure 10. The inputs to this sub-process come from the Data Collection Sub-Process 

outputs e.g., Actigraphs with recorded data and completed surveys and logs. 

27 





/ Input data \ 
into Excel ]— p- 

Excel 

1 Clean \ 

"Cleaned" Excel 
spreadsheet 
with errors 



Prepare for 
Analysis 

1 Spreadsheet j 

Spreadsheet 

\ Data 1 * 

noted in text 
file 

/ Input aata \ 

Excel 




\ 


1 into Exc&l J—► 

1 Spreadsheet j 

Spreadsheet 




Clean \ 


/ Actigraph \ 

Cleaned 

4 Data —1 

watch fles 

\ Using J 1—^ 

and Sleep 

XAction / 

statistics 

^ ^ 

report 


Analyze 

using 

FAST 

tool 


Input data 
Into Excel 
Spreadsheet 


Figure 10. Prepare Data for Analysis 


The first activities of this sub-process include transferring the data from the 
Actigraphs and paper recordings to an electronic format that allows the researcher to then 
“clean” the raw data. The data from the paper surveys and logs must be manually input 
to a Microsoft (MS) Excel spreadsheet. The data from the sleep watches has to be 
downloaded to a computer using the Actigraph, the Actigraph Interface Unit, and Act 
Millennium software. Data from each Actigraph are saved with the file extensions .DAT 
(Act Millennium Data format) and .AMI (AMI without Application format) for use with 
the Action W software. Once these raw data files have been established, the next 
activities include “cleaning” the data. 

The data from surveys and logs have to be checked manually for data entry errors. 
Additionally, the quality of the logs maintained by the participants is determined by the 
researchers. The participants often forget to record their data or become so overwhelmed 
with operational events that their logs are often incomplete or inaccurately filled out. If 
there are too many errors and they cannot be rectified, researchers may have to remove an 
entire participant’s data set from the study. Additionally, the tedious nature of manually 
entering data from paper journals to electronic media often requires the research team to 
review the data entries several times to ensure accuracy. 

The Actigraph raw data are imported to the Action W software. Based on the 
algorithm for sleep scoring activity level. Action W automatically codes the Actigraph’s 
recorded activity epochs as sleep or wake episodes. An example of raw data of a 
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participant that has been automatically coded by Action W software is shown in Figure 
11. The coded data are compared with the sleep/wake log of the participant. If there are 
discrepancies, the researcher has to determine the best code for the data. This 
subjectivity is necessary to overcome limitations of the equipment. 
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Figure 11. Automatically Scored WAM Data Using Action W 


In this example, the participant logged that he had taken off his WAM for a 
shower during the times circled in Figure 12. Because the activity levels are at or near 
zero, the Action W scores these periods as sleep epochs (indicated by the red line beneath 
the recorded time periods). 
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Figure 12. Circled Times the Participant Removed His Actigraph for Showers 
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The researcher can manually recode these epochs as sleep, as shown in Figure 13, 
to allow more a more accurate depiction of the participant’s behavior. Once “cleaned,” 
the file is saved with file extensions .AMI (for reuse with Action W) and as .EBE (epoch 
by epoch for use with the EAST software). Any data that had been corrupted due to 
mechanical or human errors are also identified and may be removed from the study 
altogether. Action W also calculates sleep statistics for each data file. The output 
products of this activity are the data files themselves, the statistics reports for each 
participant and the MS Excel spreadsheets with recorded survey and log data. 
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Figure 13. Manually Recoded Shower Times as Wake Epochs 


The Actigraph data are then processed in two ways. Eirst, the researcher exports 
the sleep statistics from the Action W data files into a MS Excel spreadsheet and 
manually manipulates these into a format compatible with data analysis tools for use at a 
later stage in the process. The second way the data are processed is using the EAST 
modeling tool. The researcher sets up a schedule encompassing the period of the study 
and pre-conditions the three previous days of the data collection period to match a 
realistic sleep/wake cycle of the participant. If enough data were collected, the researcher 
can use the first three days of collected data to automatically set the pre-conditions for the 
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participant. Once a FAST schedule is defined, the researcher imports the .EBE Actigraph 
file of each participant to this schedule and is able to model the performance 
effectiveness during the period of the study. The researcher compares the EAST analysis 
to the logs and surveys to identify correlations. Any required data from the EAST models 
are recorded in MS Excel spreadsheets to allow for further data analysis by the 
researcher. The outputs of this sub-process include all Action W sleep/wake periods 
screen captures, EAST performance effectiveness model screen captures, as well as the 
“cleaned” MS Excel spreadsheets prepared for export to data analysis tools. 

3. Analyze and Publish Sub-Process 

The data from the previous sub-process are analyzed and the comprehensive analysis is 
published in the Analyze and Publish Sub-Process shown in Eigure 14. The “cleaned” 
data are analyzed using MS Excel tools as well as exported to SPSS (statistical software 
package used by HSIE researchers) for further analysis. Once in SPSS, the data have to 
be properly formatted for analysis. The output of the analysis includes the results of 
various statistical tests performed on the data in the form of tables and graphs. The 
research team then combines all of their efforts and writes their thesis using the outputs 
of these analysis tools and the outputs from other sub-processes. The final output of the 
whole fatigue study process is the published thesis where conclusions are drawn based on 
the analysis conducted. 



Figure 14. Analyze and Publish Sub-Process 


E. RETURN ON KNOWLEDGE FOR “AS-IS” PROCESS USING KVA 
ANALYSIS 

In order to compare the current process with any newly engineered process, a 
framework of comparison has to be established. Using the KVA methodology, we are 
able to calculate a ROK (Return on Knowledge) of the current sub-processes in terms that 
we can compare with the reengineered process. Additionally, KVA allows us to identify 

the shortcomings of the current process. The numerator in the ROK ratio is the total 
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amount of learning time required to execute a process. The denominator is the cost of 
executing the process (or firing that knowledge). A summary of the core process and 
sub-processes identified in STEP 1 are shown in Figure 15. Following STEP 2, the 
common units used to calculate for learning time in hours were established. The number 
of hours required to learn to execute each task was calculated as part of STEP 3. Where 
applicable, the learning time of associated IT products when used in the execution of the 
task was also added to the total learning time. Following STEP 4, the representative 
sampling time period was determined to be one year. To determine the weighted learning 
time for each sub-process according to STEP 5, the learning time required for one 
execution of each task was determined through interviews with SMEs. 





Figure 15. “AS-IS” Process 


Some tasks were executed more than one time. For example, when preparing 
equipment for the study, the task of initializing the Actigraph watch had to be repeated 
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for every participant on the study. If the task was executed more than one time, then the 
knowledge for that task fired more than one time. The number of times the knowledge 
was fired in the given sample period is referred to as the weighted learning time. In the 
NPS studies, the average number of studies completed in each sample period of one year 
was one and the average number of participants identified for a study was 25. However, 
due to human and equipment errors, the SMEs estimated that on average 20% of data 
from the participants were not usable in the study. In other words, the average number of 
participants with adequate data was estimated to be 20 per study. The weighted learning 
time is only calculated for the amount of successful executions of the tasks. Therefore, in 
a task such as initializing the Actigraphs, the learning time for one execution of the task 
was multiplied by 20 (the number of successful executions) to get the weighted learning 
time for the task. After all of the weighted learning times for each task were calculated, 
they were added together for a cumulative learning time in hours for the activity. 
Correspondingly, the total learning times for each of the activities were added together 
representing the total learning time for the sub-process. In STEP 6, the cost to execute 
each task was calculated. The hourly wage of the researcher was conservatively 
estimated at $22 (based on the annual base pay of an 0-3 in the active service). The 
cycle time or the amount of time for the researcher to execute the task was calculated. 
Because the cost included the total number of times the task was executed, the cost for 
the task included the times the task was executed but the data was not used. Eor example, 
if the researcher had to initialize 25 Actigraphs, the cost calculated for the task was the 
cycle time to initialize an Actigraph multiplied by 25 watches. After the cost for each 
task was determined, the total cost for each activity was calculated by adding the cost for 
each task. Similarly, the total cost for each sub-process was then calculated by adding 
the total costs of each activity included in this sub-process. To determine the ROK 
according to STEP 7, the ratio of weighted learning time divided by cost was calculated. 
A summary of the calculated weighted learning times and total costs to produce the ROK 
for each sub-process is shown in Table 1. (Eor a comprehensive table of calculations by 
task, see Appendix A.) At NPS, these research studies are conducted by students 
working under a principle investigator/professor. As a result, the ability to maintain 
corporate knowledge is extremely difficult. Once a student becomes familiar with how to 
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complete the tasks, he graduates resulting in a loss of knowledge he acquired while 
executing the tasks for the study. Because of this major factor, the ROK of the whole 
process is inherently low. The reengineering effort is critical to try to streamline the 
process to make it as efficient as possible but also to try to capture some of the 
knowledge in IT tools that allow NFS to maintain some of the corporate knowledge they 
are continually losing. It is imperative to find a reengineered process that will cost less 
and provide greater return. 


Sub Processes 

Weighted 

Learning 

Time 

(Hrs) 

Total 
Cost ($) 

As-Is 
ROK 

Collect Data 

289 

3300 

8.76% 

Prepare for Analysis 

1158 

8277.5 

13.99% 

Analyze and Publish 

1161 

10192 

11.39% 





TOTAL 

2608 

21769.5 

11.98% 


Table 1. “AS-IS” ROK 


F. AREAS TO FOCUS REENGINEERING EFFORT 

Looking at the ROK for each sub-process at the activity and task level, some tasks 
were clearly identified as providing low return on knowledge. After consultation with the 
SMEs, the data collection and data storage areas of the process caused the greatest 
concern. The data collection methods of the “AS-IS” process are tedious and error prone. 
The paper logs and surveys are a hindrance to participants based on exit surveys from the 
recent fatigue studies. The paper entries are a source of a tremendous amount of 
additional work. Requiring the participants to enter the data once on paper and then the 
researcher to transfer the same data to a spreadsheet is an example of the inefficiency 
inherent in this process. The extra time required to diligently “clean” the data by 
comparing the paper entries to the computer entries demonstrates little Return on 
Knowledge. The conservative estimate of a 20% loss of collected data due to the 
combination of human and mechanical errors appears unacceptably high and avoidable. 
Additionally, the lack of data storage capability is a significant drawback of the current 
process. While Dr. Miller is involved as an advisor in all of the studies conducted by 
HSIL, each researcher determines how he saves his data for data analysis. Because of the 
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disparate ways of storage, the effective result is that the published thesis is the only 
source of storage of collected data. To use this for future studies or conduct any meta¬ 
analysis is virtually impossible given the cost required to do so. Considering the 
particular areas of low ROK in the “AS-IS” model, Figure 16 highlights the particular 
activities targeted in the reengineering effort. 

After analyzing the ROK for each task and discussing the complexities of each 
with the subject matter experts, the focus of the reengineering effort became clear: 
streamlining the data collection process and storing data for future use. 
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Figure 16. Areas of Focus for Reengineering Effort 
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IV. REENGINEERING THE PROCESS 


A. OVERVIEW 

The reengineering effort of this thesis centered around creating a data repository 
and streamlining the data collection methods using current information technology. The 
research teams that conduct the human performance and fatigue research at NFS are 
typically thesis students working towards a Master’s Degree in Operations Research. 
While their knowledge of data analysis software is extensive, their experience with web¬ 
page design, web servers and databases is often limited. Therefore, the technology 
inserted into the “TO-BE” process had to be relatively easy for the researchers to learn 
while still maintaining the desired effect of streamlining the process. Also, the software 
and hardware we selected for any IT implementation had to be easily supportable. We 
did not want the Human Systems Integration Lab to become dependent on a specific 
vendor where costs to maintain the software became unreasonably high or proprietary 
issues would cause expensive upgrades. Every attempt was made to use IT assets that 
were either non-proprietary or that was supported by the IT staff at NFS. This chapter 
includes a consolidation of the Flanning, Requirement Analysis, Design and 
Implementation Fhases of the Study Repository Template database system development 
as well as the data collection technique development. A summary of the database design 
created in Microsoft Access 2002 and the collection methods including the automated 
daily log created using HanDBase 3.0 and the web-based survey forms created using 
Microsoft ErontFage 2002 are included in the Implementation Fhase. 

B. PLANNING (SURVEY) PHASE 

After extensive study of the “AS-IS” process, we targeted our reengineering 
efforts to achieve two main goals. First, we wanted to allow participants and researchers 
to be able to easily record data with simple and standard data collection methods. 
Secondly, we wanted to provide the HSIL with a data storage capability that brought 
standardization to the fatigue studies. Along with both of these goals, we also wanted to 
afford researchers the ability to easily adapt the collection and storage means to suit the 
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unique requirement demands for each study. In the planning phase, we refined these 
goals to prepare for the design and implementation of IT solutions to enhance the “TO- 
BE” process. 

1. Problem and Scope 

Our challenge was to develop a database and data collection techniques to 
facilitate efficient and timely data analysis for operational fatigue studies in an automated 
setting which can be implemented by future fatigue research teams from the HSIL at 
NPS. Addressing these issues, we used a three tiered architecture approach to data 
collection and storage in the reengineering process: a user interface layer, a database 
layer and a web-server layer. The graphical user interface layer allows data to be entered 
into and extracted from the database layer. Because both study participants and 
researchers required access to the databases, it was important to keep the graphical user 
interfaces simple and self explanatory. By automating the data entry, the research teams 
can eliminate the task of transposing data from paper surveys to electronic spreadsheets 
thus reducing the number of errors. The database layer provides flexibility and efficiency 
for data storage. Additionally, a template database provides the HSIL researchers with a 
method of standardization to maintain data from all fatigue studies that currently does not 
exist. The web-server enables participants to have the ability to enter data using the web 
while giving researchers the ability to access data from many studies from a single 
storage area. Templates of survey designs from data collection techniques as well as the 
database template stored on the web-server provide institutional knowledge from which 
each research team can build and subsequently add. The following description 
summarizes the automated functions required of the system. The system: 

• Enables a research team to record the parameters of a study including 
information such as duration, location and a summary of the study; 

• Allows participants to enter survey data via the web prior to the start of the 
study to allow the research team to prepare necessary equipment and 
paperwork for the study ahead of time as well as conduct preliminary 
analysis of participants prior to the study; 

• Eacilitates the preparation of the equipment by enabling research teams the 
ability to maintain calibration and initialization status of the equipment as 
it is prepared for use in a field study; 
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• Provides the capability for the researcher to assign equipment to 
participants by serial number and track status of collected equipment and 
data after the data collection portion of the study is finished; 

• Enables participants to log daily events easily without hindering them in 
their operational settings; 

• Provides a means to collect performance testing data from participants in 
an unobtrusive manner; 

• Provides the flexibility to easily change or expand data collection 
requirements to accommodate researchers when different or additional 
data variables must be collected; and 

• Supplies the HSIL research teams at NPS with standardized templates for 
data collection and storage methods, expanding and retaining their 
corporate knowledge. 

2. Constraints 

Though each study conducted by HSIL has unique requirements, the prototype 
system is designed as a template which a research team can easily tailor to suit their 
particular study’s requirements. The participant surveys for this system include a 
Demographic Survey template providing information about the individual participant and 
a Lark and Owl Survey based on the momingness/eveningness scale included in 
APPENDIX B. The template also includes the capability to collect performance data on 
participants using the ARES software designed for use on the PDA. Einally, an event log 
template was created from a software database development package. 

C. REQUIREMENTS ANALYSIS PHASE 

1. Methods of Data Collection 

Survey input forms were required to collect survey data from participants via the 
web. Many web-page design software packages were evaluated including non¬ 
proprietary software. Because the research teams from the HSIL generally have little to 
no web design skills, it was important to choose software that was relatively simple to use 
and supported by NPS IT staff. Microsoft IrontPage is easy to use and classes are 
routinely offered by NPS IT staff for Microsoft ErontPage. Also, the software is 
available through the NPS academic use license so no additional costs for the HSIL are 
incurred. The template design includes a generic Demographic Survey and a 
standardized Lark and Owl Survey. 
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To maintain equipment statuses and to simplify the assignment of equipment to 
participants, data input forms for the researchers to use were also required. In addition, 
the SMEs articulated a tracking requirement for maintaining a record of when assigned 
equipment from a participant was collected and when data from a participant had been 
successfully downloaded to the database. A form allowing researchers to track their 
work progress was needed to fulfill this requirement. Because of the pressures on 
researchers conducting a field study, an effort was made to standardize the data collection 
processes to alleviate some of their administrative burdens. 

In order to efficiently “clean” participants’ Actigraph data and analyze 
participants’ behavior with respect to their work schedules, it is imperative for 
researchers to be able to view a daily event log from participants. The information 
collected on this log includes events such as when a participant removed his Actigraph, 
when he went down for sleep, when he awoke and when he started and ended a work 
shift. Because these events are logged throughout the day, using a PDA to record these 
events provides an unobtrusive means for data collection. 

ARES performance testing software for PDAs was used. The data collected from 
each test a participant takes are stored in a file on the PDA and can be downloaded 
directly into the accompanying Data Manager software for ARES data or stored on the 
PC using the HotSync function and later imported to the Data Manager software. Once 
opened in Data Manager, the data can be copied to a MS Excel spreadsheet for analysis. 

2. Conceptual Data Model for the Database 

After the thorough analysis of the current process, we gained a comprehensive 
understanding of the data required for collection and the processes involved in using 
particular data for analysis. Through SME interviews, we also gleaned information 

regarding the potential future uses for the data to ensure we incorporated into the storage 
facility all pertinent information for a particular study. The subsequent paragraphs 
describe the requirements for the database design. 
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a. Entity Description 

After clarifying the data requirements, the following entities were 
modeled: STUDY, RESEARCH TEAM, PARTICIPANT, ACTIGRAPH, PDA, EOG, 
ARES DATA, EARK AND OWE, ACTGRAPH STATUS, PDA STATUS, and 
COEEECTED EQUIPMENT AND DATA. The attributes for each of these were clearly 
defined. The ARES DATA attributes include those defined by the ARES performance 
testing software designed for the PDA. The attributes for the EARK AND OWE are 
based on the standardized morningness/eveningness scale in Appendix B. The following 
lists these attributes for each entity: 


41 



PARTICIPANT 

Study ID 

ParticipantID 

LastName 

FirstName 

Middlelnitial 

Rank 

Prior 

Specialty 

EmailAddress 

PhoneNumber 

DSN 

PagerNumber 

OfficeAddress 

City 

State 

Age 

Gender 

STUDY 

Study ID 
Study_Name 
Study_Description 
S tudy_B egin_Date 
S tudy_End_D ate 
Study _Duration 
Study _Location 
Number_Participants 


ARES DATA* 

Record 

Subject *NOTE: Tbe list of attributes for ARES 

Test is truncated. In tbe completed 

Session database, all attributes as they appear 

PIndex in tbe DataMan software package 

Rep are included. 

TestVer 

Battery 

BatVer 

AresVer 

Seed 

Start 

Dur 

Stim 

Cor 

Err 

Imp 

Lap 

ITI 

PDA STATUS 
PDA ID 
ARES Loaded 
HanDBase Loaded 
Log Database Loaded 


ACTIGRAPH 

ActigrapbID 

ParticipantID 

Lastname 

PDA 

PDAID 

ParticipantID 

Lastname 

LOG 

RecordID 

ParticipantID 

Date 

Time 

Event 

Note 


RESEARCH TEAM 

ResearcberlD 

LastName 

FirstName 

Middlelnitial 

Title_Rank 

EmailAddress 

PboneNumber 

PagerNumber 

OfficeAddress 

City 

State 

LARK AND OWL 

Lark and Owl ID 
Participant ID 
LastName 

Up_later_tban_planned 

Bed_T ime_N o_Commitments 

Early_Exercise 

Evening_Tiredness 

Cboose_Own_Working_Hours 

Self_Consideration 

Total 

Lark or Owl 

ACTIGRAPH STATUS 

Actigrapb ID 
Battery Last Changed 
Times tamp 


COLLECTED EQUIPMENT AND DATA 

Lastname 
ParticipantID 
Consent Forms Signed 
Collected Actigrapb 
Collected PDA 
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b. Database Perspectives 

The entities can be described using a main success scenario as follows: 

• A request to the HSIL at NFS for a fatigue study defines the study 
parameters. The RESEARCH TEAM is formed and contact information 
is collected on them. The researchers then record the STUDY parameters 
and coordinate with the sponsoring organization to identify 
PARTICIPANTS. 

• PARTICIPANTS complete the Demographic Survey and the EARK AND 
OWE survey via web input forms. 

• The RESEARCH TEAM updates the ACTIGRAPH STATUS and PDA 
STATUS and assigns ACTIGRAPHs and PDAs to PARTICIPANTS. 

• The RESEARCH TEAM prepares necessary consent forms and bundles 
equipment for issue. Once at the location of the STUDY, the RESEARCH 
TEAM issues the equipment (ACTIGRAPHs and PDAs). 

• PARTICIPANTS wear the ACTIGRAPH watches and use the PDA to 
EOG daily events and to take ARES performance tests according to the 
prescribed schedule. 

• The RESEARCH TEAM collects the equipment and downloads the 
ACTIGRAPH data using the Act Millennium software. They also 
download the ARES DATA and the EOG data using the HotSync function 
on the PDAs and then copy this data to the ARES DATA and EOG tables. 
The COEEECTED EQUIPMENT AND DATA are updated to ensure all 
equipment is accounted for and all available data have been collected from 
the PDAs as well as the ACTIGRAPHs for each PARTICIPANT. 

• A EOG report is generated to simplify the coding of actigraphy and the 
analysis of PARTICIPANT behavior and performance. 

c. Design Methodology 

Through analysis of the current process, we were able to determine which 
data researchers sought to gain from the data collection phase in order to complete their 
data analysis. Our goal was to reverse-engineer the database system from a bottom-up 
perspective. We collected potential attributes including those already defined by surveys, 
the PDA log and performance testing software, linked them to entities, and then worked 
to resolve the relationships between them. 

3. Process Data Model 

Using web forms generated with Microsoft ErontPage, the PDA Event Eog, PDA 
ARES testing software as well as forms and reports within the DBMS system, the 
following are the system components. 
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a. Data Input 

• Create/modify/delete a RESEARCH TEAM. 

• Create/modify/delete a STUDY. 

• Create/modify/delete a PARTICIPANT. 

• Create/modify/delete EOG events. 

• Create/modify/delete ARES data. 

• Track ACTIGRAPH STATUS. 

• Track PDA STATUS. 

• Track COEEECTED EQUIPMENT AND DATA. 

b. Data Maintenance 

• Update ACTIGRAPH STATUS. 

• Update PDA STATUS. 

• ASSIGN PDAs. 

• ASSIGN ACTIGRAPHs. 

• Update COEEECTED EQUIPMENT AND DATA. 

c. Reports 

• Prepare Participant Information Report to assist Research Team with 
preliminary data analysis. 

• Prepare Assigned Actigraph and Assigned PDA Reports to assist Research 
Team in tracking equipment. 

• Prepare Actigraph Sign-Out Report to assist Research Team with 
maintaining property accountability. 

• Prepare formatted Eog Report to facilitate “cleaning” Actigraph data and 
assist Research Team with data analysis. 

D. DESIGN PHASE 

1. Designing Data Collection Methods 

a. Web Survey Designs 

Securing space on the HSIE web-server, we created and saved survey 
templates allowing researchers the ability to make modifications to suit each study’s 
requirements. We designed a generic Demographic Survey Eorm and a Eark and Owl 
Eorm using MS ErontPage 2002. Portions of each of these templates are shown in Eigure 
17 and Eigure 18. The forms in their entirety are shown in Appendix B. MS ErontPage 
2002 allowed is to save the results of the survey to a database on the server. The two 
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databases based on these two surveys, Demographic and Lark and Owl databases, would 
accumulate the survey results from each participant’s submission of the data. Because 
the storage database is also on the server, we were able to update this survey information 
into the main Study Repository Template Database with a simple update query function 
added to the start up of the database. 
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Figure 17. Demographic Survey Template Form 
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Lark and Owl 

Instructions: 

Take the following survey to help us determine if you are a lark, and owl or somewhere in 
between. These questions should be answered as if you were at home and not deployed? 

Last Name I Type your last name 

Circadian Identity: 

1. During your last vacation week, how often did you get up later than planned or have 
difficulty getting ready on time even though you went to bed at your regular time? 

^ It never happened 

^ It happened once 

^ It happened two or more times 

^ It happened more than three times 

2. When you have no commitments the next day, at what time do you go to bed compared 
with your usual time? 

^ Seldom or never later 

Figure 18. Lark and Owl Template Form 

As indicated earlier, if the web is not aeeessible to the partieipants, eopies 
of these forms (shown in Appendix D) are ineluded in the MS Aeeess Study Repository 
Terrplate Database. 

b. Log Database Design 

After careful investigation of existing PDA logging programs, no logging 
software for PDAs was available to reeord the types of events SMEs had artieulated for 
the daily event log. Though the log was easy to design using MS Aeeess, there was no 
way to export this desktop version to an applieation available for the PDAs. The NPS IT 
staff does not eurrently have a site lieense for any PDA database development software. 
Programming applieations for PDA use was beyond the seope of our expertise. After 
evaluating several packages available on the commercial market as well as freeware, we 
ehose HanDBase 3.0 for our PDA database development. This program provides the 
capability to create and modify databases from both a desktop and PDA platform. 
HanDBase offered enough funetionality and teehnieal support to meet study 
requirements. We created a simple log database template using HanDBase software 
which is fairly easy to modify. Because no site licensing was available for HanDBase 
3.0, a license is required for every PDA. 
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Figure 19. PDA Event Log Template 


Loaded on each PDA, the log database enables a participant to record an 
event, the date and time of the event and a short note if clarification is required. The 
events we included in the template are generic and can easily be modified to include 
particular events the researchers would like to capture for a given study. A screen 
capture from the PDA of the Event Log Template with sample data is shown in Figure 
19. The recorded data are stored on a file in the PDA and can be downloaded to a PC 
using the HotSync function of the PDA. The HanDBase software includes a desktop 
version allowing the data from each PDA to be easily consolidated and exported to a MS 
Excel spreadsheet. 

c. Collecting ARES (ANAAf^ Readiness Evaluation System) Data 
A collection means for performance data is available on the PDA using 
ARES software shown in Figure 20. Researchers generally require participants to 
complete a designated ARES battery of tests (e.g., reaction time, running memory, and 
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sleepiness scale) at least three times a day. Data are recorded for each test and are stored 
into a PDA database file. When the PDA performs a HotSync function with the laptop, 
these 


AKIS 
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AM AM ; 

E'- iili.J 



RRES b a series of simple tests 
of mental functioa 


Log In <295 days left) 




Figure 20. ARES Software for the PDA 


d. Research Team Input Forms 

Data are saved to a database file which is able to be read only by the Data Manager 
software. 

Figure 21 shows the raw data in Data Manager software. From Data 
Manager, the researcher can use the copy function to copy the data to a MS Excel 
spreadsheet for analysis (shown in Figure 22). 
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Figure 21. Raw ARES Data in Data Manager Software 
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Figure 22. Raw ARES Data Copied to MS Access Database 


Other data maintained by research teams will also be collected 
electronically. Information on the research team and study parameters will be recorded 
by the researchers using automated input forms shown in Figure 23 and Figure 24 
respectively. Additionally, researchers will maintain and update equipment statuses, 
assign equipment to participants and record status of collection of data and equipment 
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from participants at the end of a study using simple input forms. All input forms 
available in the Study Repository Template Database are also shown in Appendix D. The 
use of these electronic forms increases the efficiency of the research team’s ability to 
conduct the study and minimizes human errors associated with tracking by pen and paper. 



Figure 23. Input Research Team Information Form 
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Figure 24. Input Study Information Form 


2. Database Design 

We chose Microsoft Access 2002 as the software to create our Study Repository 
Template Database because it provided adequate support for small relational database 
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development. It is fairly easy to use and it is supported by NPS IT staff. There were no 
additional licensing requirements to use it and there is a fairly extensive MS Access 
knowledge base at NPS to assist research teams with database modifications. We 
examined the feasibility of using the latest version of Extensible Markup Language 
(XML) developed and supported by the World Wide Web Consortium (W3C) to create 
the database to avoid proprietary issues and created a prototype database using XML Spy 
software. While there are many benefits to using XML, the current W3C version of 
XML does not easily support relational databases and the complexity involved in 
maintaining referential integrity for our database proved too complex. According to the 
W3C discussion groups, future versions of XML will provide more capabilities to 
develop relational databases. Microsoft has created a version of XML (MSXML) which 
supports relational database development; however, MSXML is not W3C compliant and 
is clearly proprietary. Additionally, MSXML is not exclusively a database development 
tool. Creating databases using MSXML requires extensive knowledge of the XML 
language as well as a thorough understanding of databases. Because of its maturity as a 
product, MS Access 2002 proved to be a more robust solution for us to use. If future 
W3C versions of XML support relational databases adequately, the Study Repository 
Template database we have created can be exported to a XML format for modification 
and maintained as an XML database for suture use. 

Transforming entities to relations, we recognized the attributes identifying each 
entity and made these the primary keys of the relations. When any of these primary keys 
were required as attribute of other relations, we depicted them as foreign keys. Ligure 25 
depicts the relationships created using MS Access 2002. The highlighted text in each 
relation indicates the primary key(s). 
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Figure 25. Study Repository Template Relationships 


3. Application Design 

The Study Repository Template Database was designed to provide research teams 
with a robust database to which minor modificatbns can be made to tailor it for a given 
study. In designing our database application, no mechanisms for limiting control were 
necessary because the research team will be the administrators of their modified 
databases. In this template design, input forms and reports were created to allow the 
research team to input data efficiently and retrieve data in useful formats for data 
analysis. A Main Menu Form was created to allow researchers to easily navigate their 
way through these useful forms and reports. 

E. IMPLEMENTATION PHASE 

1. Creating the Database 

We published the template web surveys, created the event log on the PDA, loaded 
ARES on the PDA and created the research team input forms according to the design. 
After these data collection methods were refined, we were able to finalize the Study 
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Repository Template Database design and create it using Microsoft Access 2002. A 
listing generated by the Access program of the database tables, their properties, and 
relationships can be found in Appendix C. In Appendix D, graphical representations of 
all forms and reports available from the Main Menu Form of the Study Repository 
Template Database are also available. This database resides on the HSIL web server 
along with the web survey forms. The Event Log and the ARES program are loaded on 
PDAs in the HSIL and the accompanying software to both of these programs is loaded on 
the desktops in the HSIL as well as the laptop that accompanies the research team to the 
location of the studies conducted. 

2. Storing Data to the Database 

At the start of a study, the research team can enter data to the RESEARCH TEAM 
and STUDY tables using the forms from the Main Menu Eorm shown in Eigure 26. 
These forms are also displayed in Appendix D. Researchers can also update the status of 
the Actigraphs and the PDAs in the ACTIGRAPH STATUS and PDA STATUS tables 
respectively, using input forms available from the same Main Menu Eorm. 
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Figure 26. Main Menu Form 
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After the participants have been identified and have completed the online surveys, 
the Study Repository Template Database automatically updates the PARTICIPANT and 
LARK AND OWL tables when the repository study database is opened with the data that 
has been entered by participants via the web and saved to the Demographic and the Lark 
and Owl databases. If web access is not available to the participants, the research team 
can allow the participants to complete the input forms created with MS Access available 
from the Main Menu Form. If the participants have no access to computers, these input 
forms can even be printed if necessary and the researcher can add the data to the database 
using the completed printed forms and the database input form to ease the transfer of data 
from paper to the database. The researchers can update the ACTIGRAPH and PDA 
tables using the Assign Actigraph and Assign PDA Forms respectively, shown in 
APPENDIX D. After the research team downloads the ARES and EOG data from PDAs 
to MS Excel spreadsheets, the datasheets are copied to the respective tables. Einally, as 
the research team collects equipment, they can use the Collect Equipment Eorm to update 
the COEEECT EQUIPMENT AND DATA table. 

Various reports are available to the research team to use during or after the study 
and are also accessible from the Main Menu Eorm. Examples of all of these reports 
available are shown in Appendix D. These were designed to assist the researchers with 
assigning and keeping track of equipment used in the study as well as to print the final 
event log. This Eog Report, shown in Eigure 27, includes conditional formatting which 
allows the research team to easily code the actigraphy data and correlate the participants’ 
effectiveness with their daily activities using EAST and ARES analysis tools. 

3. Application Implementation 

Though the research team will modify the design template to suit the individual 
study requirements, the Main Menu Eorm allows the researcher to easily navigate 
through the database updating information and generating valuable reports as needed. 
This Main Menu Eorm can also be modified to accommodate the database changes made 
for a particular study or to add more useful reports required for the research being 
conducted. 
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Figure 27. Log Event Report 


When this template design was created, we assumed participants would have 
access to the web to provide survey information. If this is not possible, researchers can 
recreate these surveys using MS Access forms. When the system is used in this capacity, 
it is important for the research team to incorporated adequate control mechanisms to 
allow participants to enter and verify data while preventing them from accessing 
restricted data. The Help feature in MS Access outlines these control features in a “how¬ 
to” format that is easy for the researchers to follow. For the forms created using MS 
Access in the database template, constraints were used to reduce the potential for data 
entry errors for the research team. Many forms include drop down lists and check boxes 
to prevent ambiguity or misrepresented data. An example of this is the Assign 
Actigraphs Form shown in Figure 28. The drop down list for the Actigraph Number is 
taken from the query based on the ACTIGRAPH STATUS Table. This not only ensures 
researchers label the numbers with the same format but also eliminates the error of 
assigning an Actigraph that has not been added to the ACTIGRAPH STATUS Table. 

Similarly, the dropdown list for the Participant’s Last Name is available from a query 
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based on the PARTICIPANT table allowing the researcher to only assign Actigraphs to 
identified participants. Also, once the name is selected, the Participant ID field is 
automatically filled with the Participant ID associated with the selected name. 



Figure 28. Assign Actigraph Input Form 

F. EVALUATING THE REENGINEERED PROCESS 

The entire application including the web Demographic and the Lark and Owl 
Surveys, the PDA Event Log, the ARES Program for the PDA, the Study Repository 
Template Database with research input forms along with the Actigraphs was packaged 
and tested. A group of volunteer students at NPS were modeled as a group of study 
participants. As the simulated research team, we asked them to take the web surveys, 
wear the Actigraph watches, take the ARES battery of tests on a set schedule and log 
their daily events with the PDA for a period of a week. We recorded study and research 
team information, updated the equipment statuses, and assigned equipment to the 
participants after they had completed the web surveys. At the conclusion of the week, we 
collected the equipment and downloaded the data from the Actigraphs and PDAs. We 
copied the ARES and Log Data to the database and conducted exit interviews with the 
participants. Making no significant modifications, we were satisfied with the prototype 
of the “TO-BE” process. 


56 



















V. REENGINEERED PROCESS 


A. OVERVIEW 

The newly created IT tools described in Chapter IV were incorporated into the 
reengineered “AS-IS” process. In this chapter, the new “TO-BE” process is outlined. 
Using the seven step KVA process, the “TO-BE” process Return on Knowledge (ROK) 
was calculated at the process and sub-process levels using very conservative estimates to 
provide the means for comparison of the “AS-IS” to the “TO-BE” process. A report of 
the evaluation of this KVA comparative analysis is also provided. At the conclusion of 
this chapter, a discussion of the potential for a dramatic increase to ROK through 
implementation of a radical redesign to the process is offered. This impact of this radical 
redesign for the Department of the Navy and possibly other services in the Department of 
Defense is worthy of consideration. 

B. “TO-BE” PROCESS MODEL 

As we conducted the test in the laboratory using the volunteer students as 
participants, we carefully reengineered the “AS-IS” process. Incorporating the newly 
designed data collection techniques and the database template, we were able to model the 
‘TO-BE” fatigue study process. Applying the KVA methodology, we determined the 
core and sub-processes for the “TO-BE” process. Though the titles of these remain the 
same as in the “AS-IS” process, many of the subordinate activities and tasks changed. A 
complete set of diagrams of these sub-processes, activities and tasks is provided with 
KVA calculations in APPENDIX A. The following paragraphs highlight the significant 
changes made and the impact they have for the efficiency and accuracy of the data 
collection and analysis conducted by research teams from the HSIE at NPS. 

1. Phase 1: Collect Data Sub-Process 

The major reengineering effort happened in this phase. Eigure 29 depicts the 
Collect Data Sub-Process of the “AS-IS” process as the baseline for comparison and 
Eigure 30 shows the Redesigned Collect Data Sub-Process of the “TO-BE” process. The 
Prepare Data for Collection Activity now includes modifying the Study Repository 
Template Database, modifying the Demographic Survey on the HSIE webpage, 
appending the Eark and Owl Survey to the webpage, and modifying the PDA Event Eog. 
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Figure 29. AS-IS Collect Data Sub-Process 



http://www.nps.navv.mil/humanfactors . The web surveys, their 
corresponding databases and the Study Repository database are all stored on the server. 
These administrative tasks take considerably less time than those under the corresponding 
“AS-IS” activity because the researchers modify existing templates rather than create 
them on their own. Additionally, by completing these tasks at this point in the process, 
the research team is prepared to collect survey data via the HSIL webpage once 
participants have been identified by the sponsoring organization of the study. This 
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participant survey data are also checked for completeness and screened for potential 
errors or issues. This pre-screening of participant data allows researchers the ability to 
prevent the 20% loss of data generally experienced using the “AS-IS” process. 

Once the participants are recorded in the database, the equipment is prepared. 
The Actigraph watches are initialized and assigned by serial number to the participants 
using the Assign Actigraph Form. The PDAs are similarly assigned by name (e.g., 
HSILl, HSIL2 etc.) to participants using the Assign PDA Form. The HanDBase 
software and the Study Event Log database file and form file are loaded onto the PDA. 
Additionally, the ARES software is loaded onto the PDAs and each participant is 
registered by participant ID and the particular battery of tests is selected. Consent forms 
are printed and all equipment and forms are bundled for issue under the Prepare 
Equipment Activity. This activity can now be completed prior to the research team ever 
arriving at the study location. A tremendous burden is lifted from the researchers by 
doing so. Now they can come to the study location and focus on collecting accurate 
actigraphy, ARES performance data and log data instead of worrying about the logistics 
of identifying participants and tracking the assignment and issue of equipment to them. 
During the collection process, researchers can check participant data for completeness by 
routinely downloading data from the Actigraphs and the PDAs. This ensures that 
equipment is working properly, participants are taking their ARES tests, and that they are 
adequately logging their daily behavior. At the end of this sub-process, the research team 
will have a record of the study parameters and the research team information as well as 
“clean” participant and Lark and Owl data loaded in their database. They will have 
PDAs and Actigraphs with data stored on them and signed participant consent forms. 
Compared with the “AS-IS” sub-process, the research team will have collected more 
accurate data and will have already completed all of the survey data preparation for 
analysis by the end of this phase in the “TO-BE” process. 

2. Phase 2: Prepare Data for Analysis Sub-Process 

Comparing the “AS-IS” Prepare Data for Analysis Sub-Process shown in Eigure 
31 to the “TO-BE” Sub-Process shown in Eigure 32, the Input Data to Excel 
Spreadsheets Activity and the Clean Data Activity are eliminated. Because the 
Demographic and Lark and Owl survey data have been collected and verified for 
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completeness in the Collect Data Sub-Process, the only data requiring preparation for 
analysis in the ‘TO-BE” Prepare Data for Analysis Sub-Process would be the Event Eog 
data, the Actigraph data, and the newly added ARES data. 
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loads the ARES Data (.PDB file) and copies the data to the ARES Data table in the Study 
Repository database. The research team can use the Data Manager software to conduct 
analysis and the data in the MS Access table can be exported easily to other data analysis 
software. The Event Eog Data are viewed with the HanDBase Desktop software and 
copied to the Eog table in the Study Repository database. Without further manipulation, 
the Eog Report is generated by clicking the “Eog Report” command button on the Main 
Menu Eorm in the database. The conditionally formatted Eog Report highlights the 
“ON/OEE” state of an event, for example Actigraph Watch ON and Actigraph Watch 
OEE. The report is generated in chronological order by participant. On the web server, a 
Study Repository folder 

At the heart of fatigue studies are the actigraphy data. The scoring algorithm used 
in the Actigraph software determines whether a sleep epoch is classified as either sleep or 
wake. Based on a participant’s log and a researcher’s subjective expertise, the Actigraph 
data nust be “cleaned.” If the software coded low activity levels as sleep when the 
participant was really awake, the participant may be wrongfully thought to have had more 
sleep and thus potential higher performance effectiveness than he actually had. To 
accurately portray a participant’s activity, the sleep/wake log is essential. Eigure 33 
shows a participant’s Actigraph data over a period of a few days. The sleep algorithm in 
the Action W Software scored the low activity levels in the circled area as sleep. The 
intermittent activity shown during this time could lead a researcher to concur with the 
Actigraph software and assume the participant was asleep. In this case, however, the 
accompanying log report, shown in Eigure 34, clearly indicates that during that time, the 
participant had removed his Actigraph watch but was not asleep. By recoding the sleep 
data as wake data, the researcher is able to more accurately represent the behavior of that 
participant. Because of the format of the Eog Report, it is easy to see the changes of state 
a participant goes through chronologically. The highlighted events allow the research 
team to easily compare the recorded actigraphy with the log report to trim the data. This 
ease of this comparison not only saves researchers many hours of trimming data but it 
also yields a more accurate assessment of the participant’s activity than the “AS-IS” 
process does using paper logs. Because the Actigraph data files can only be read by 
Action W and EAST software in particular data formats (.AMI and .EBE respectively), it 
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is not useful to translate these files into MS Access formats. Instead, the Actigraph and 
FAST files (both raw and “cleaned”) are saved to the Study Repository folder on the 
server. 
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Figure 34. Log Report Data for Participant 01 


The most significant change to the second phase of the process is that the 
researchers do not have to transpose data from paper to electronic format. Not only does 
this save the time of entering the data into spreadsheets but more significantly, it also 
eliminates the countless hours spent comparing the spreadsheets with the original paper 
entries to verify the data accuracy. Secondly, another tremendous benefit of this 
redesigned sub-process is the amount of time saved in “cleaning” the Actigraph data 
using the Log Report generated from the raw Event Log data. Next, the standardized 
format included with the data collection and storage templates decreases the amount of 
time researchers spend formatting and storing their data. Finally, the addition of the 
ARES data provides a valuable performance measure for the research team while 
requiring minimal time to download the data using the HotSync function and to copy it to 
the MS Access database. In total, this “TO-BE” Redesigned Prepare Data for Analysis 
Sub-Process will take less than half the cycle time to complete than the “AS-IS” Prepare 
Data for Analysis Sub-Process while providing the research team with more accurate data 
to analyze. 
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3. Phase 3: Analyze and Publish Sub-Process 

The data analysis piece of the third phase of the “TO-BE” fatigue study process is 
very similar to that of the ‘AS-IS” process. The thesis output of the sub-process is the 
same. The major difference is the inclusion of the Study Repository database as the 
means of data storage available for reuse in the “TO-BE” process. Eigure 35 shows the 
“AS-IS” Analyze and Publish Sub-Process. The “TO-BE” Redesigned Analyze Data, 
Publish Analysis and Store Study Data Sub-Process depicted in Eigure 36 shows the 
addition of Clean MS Access Database to the Write Thesis Activity resulting in the new 
Study Repository folder including the database and complimentary data files as an output 
product of the sub-process. 


Analyze and 
Publish 



Figure 35. AS-IS Analyze and Publish Sub-Process 
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analysis and does not have to waste time designing a storage method. The Study 
Repository database allows them the option to add tables for analyzed data results and to 
add or modify reports for inclusion in the published documentation of the study. MS 
Access empowers the researcher with flexibility to efficiently utilize the collected data. 
Additionally, the standard format allows those interested in the fatigue research of the 
HSIL at NFS to more easily access and interpret the collected, analyzed and stored data 
of the operational fatigue studies. 

C. RETURN ON KNOWLEDGE FOR “TO-BE” PROCESS USING KVA 
ANALYSIS 

To calculate a ROK (Return on Knowledge) of the newly redesigned sub- 
processes in order to compare it with the current process, we used the seven step KVA 
methodology. A summary of the reengineered “TO-BE” core process and sub-processes 
identified in STEP 1 are shown in Eigure 37. 
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We kept the same units of hours to calculate learning time as we did in STEP 2 
for the “AS-IS” analysis. In STEP 3, the learning time for each task was calculated. The 
sampling time period of one year remained the same as was used for the “AS-IS” for 
STEP 4. In STEP 5, we calculated the weighted learning time for each sub-process. Eor 
the tasks that remained the same, we carried their learning times from the “AS-IS” 
process. Eor the newly incorporated tasks, we used the expertise of the SMEs and our 
own to determine the learning time. Because of the dramatic reduction in cycle time for 
the entire process, we conservatively estimated that the number of studies conducted each 
year could easily double in number. Additionally, the less obtrusive means of collection 
will most likely allow for an increase in the average number of participants per study. As 
a conservative measure, we kept 25 as the average number of participants per study for 
our calculations. However, due to the streamlined efficiency of collecting and cleaning 
survey data from participants prior to arriving on site, we maintain that the research team 
will be able to collect useable data on all 25 participants. When calculating the weighted 
learning time of a task, we multiplied the learning time of a task (included associated IT 
learning time if applicable) with the number of times the task was executed in a given 
sample period. Since the average number of participants is 25 and the number of studies 
conducted per year increased to two, the weighted learning time for a task such as 
initializing Actigraphs would be the total learning time multiplied by 50. Cumulative 
weighted learning times were calculated for the process and sub-process level. Similarly 
in STEP 6, the cost to execute each task was calculated as well as the cumulative costs of 
each sub-process. To determine the ROK according to STEP 7, the ratio of weighted 
learning time divided by cost was calculated. A comprehensive table of calculations by 
task is included in Appendix A. 

A comparative summary of the calculated ROK between the “AS-IS” process and 
the ‘TO-BE” process is shown in Table 2. Though the estimations were conservative, the 
ROK of the “TO-BE” process more than doubles that of the “AS-IS” process. The 
efficient data collection means of the “TO-BE” process clearly enables the research teams 
to analyze more data in a timely and efficient manner producing more accurate results. 

The bottoms-up approach we used of evaluating KVA from the task level to the 

entire process captured with granularity the increased ROK. 
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Sub Processes 

Weighted 
Learning Time 
(Hrs) 

Total Cost 

($) 

As-Is 

ROK 

To-Be 

ROK 

Collect Data 

289 

3300 

9 % 



1810 

4246 


43% 

Prepare for Analysis 

1158 

8278 

14% 



3075 

9185 


33% 

Analyze and Publish 

1161 

10192 

11% 



6885 

32082 


21% 






TOTAL 

2608 

21770 

12% 



11770 

45513 


26% 


Table 2. Comparison of “AS -IS” ROK to “TO-BE” ROK 


Additional benefits of the reengineered proeess that have yet to be realized. 
Perhaps the most notable of these is the addition of the data repository for all the HSIL 
studies eondueted at NPS whieh will enable researehers to eonduet meta-analysis aeross 
many studies. Additionally, the standardization of the “TO-BE” proeess and the ability 
to save methods used for eaeh study eondueted may allow an even greater number of 
studies to be eondueted per year the pool of resourees available to thesis student fatigue 
researehers beeomes rieher with every eompleted study. 

D. IMPACT OF A DRAMATIC PROCESS REENGINEERING EFFORT 

The goal of the fatigue researeh eondueted today and in the near-term future at the 
HSIL at NPS is to evaluate partieipant data and provide feedbaek to eommanders on the 
effeets of fatigue as it impaets human performanee effeetiveness. Migrating from the 
“AS-IS” proeess to the “TO-BE” proeess will eertainly allow researehers to aehieve this 
goal more effieiently and with greater volume potentially allowing the ability to generate 
more representative data. However, what if the goal was expanded to provide 
eommanders with near real time assessments of fatigue and performance effectiveness to 
allow them to make smarter decisions? 

By further dramatic process reengineering, this goal could be realized. Instead of 
focusing research on determining past performance effectiveness, the HSIL at NPS could 
provide commanders with more predictive tools to assess current conditions and risks 
associated with fatigue and provide the commanders with strategies to mitigate given 
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circumstances to increase this effectiveness for a particular mission. By packaging the 
data collection process into “How-To” type fatigue assessment kits issued to each ship, 
sailors and other service members could be assessed in their operational environments. 
Data collected aboard the ship by trained members of the crew could be sent 
electronically to the team of researchers at the HSIL where they would conduct their 
analysis and provide feedback to the commanders. A rough estimate of the ROK for the 
Dramatic Redesign was calculated and is shown in Appendix A. A summary of the ROK 
results compared with the “AS-IS” and “TO-BE” processes are shown in Table 3. 


Sub Processes 

Weighted 

Learning 

Time 

Total 

Cost 

As-Is 

ROK 

To-Be 

ROK 

Dramatic 

ROK 

Collect Data 

289 

3300 

9% 




1810 

4246 


43% 



395760 

370425 



107% 

Prepare for Analysis 

1158 

8278 

14% 




3075 

9185 


33% 



1204500 

2062500 



58% 

Analyze and Publish 

1161 

10192 

11% 




6885 

32082 


21% 



254250 

1122413 



23% 







TOTAL 

2608 

21770 

12% 




11770 

45513 


26% 



1854510 

3555338 



52% 


Table 3. Comparison of “Dramatic” ROK to “AS-IS” and “TO-BE” ROK 


A repository of successfully mitigated obstacles could be retained and shared as a 
“Lessons Learned” type of tool for leaders to use. The staff of researchers at the HSIL 
would certainly have to increase and could not be limited to thesis students but the 
incredible return provided to the entire Navy could positively impact the success of future 
missions and reduce number of accidents and other incidents caused by human error due 
to fatigue. 
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VI. NAVAL OFFICER INDOCTRINATION SCHOOL STUDY 


A. OVERVIEW 

The reengineered proeess was validated using partieipants from the Naval Offieer 
Indoctrination School (OIS) in Newport, RI (Figure 38). To accomplish the validation, 
we designed a study to emulate the NFS Fatigue Study Process. Because the objective of 
our study was to evaluate the “TO-BE” process, we did not seek to answer any particular 
fatigue related questions with this study. Instead, using our newly designed templates, 
we wanted to determine if data could be collected, manipulated and stored more easily 
and with less data corruption. 



Figure 38. Naval Officer Indoctrination School 


We selected the Newport location primarily because the daily routine of OIS 
students is highly regimented. This regimentation allowed us to more easily monitor the 
data collection process and to address issues as they arose. Additionally, the staff and 
faculty at OIS understood our intent and graciously allowed us to evaluate our process at 
their facility. Although the reengineering process was the focus of our study, we did 
collect human fatigue and performance data on students and faculty at OIS. These 
actigraphy and ARES performance data were collected on 20 participants from 31 March 
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2003 to 5 April 2003. We conducted only preliminary data analysis to evaluate the 
changes we had incorporated into the “TO-BE” process, however, we did not conduct an 
in depth analysis of the data. Since the preliminary analysis revealed interesting 
observations with regard to human fatigue and performance, we have made our OIS 
Study data available for further analysis by fatigue researchers on the HSIL web server 
and it is anticipated that NFS students will use the data for further analyses. 

Broken into phases, this chapter provides a summary of the OIS Study followed 
by lessons learned. Using the templates created in the reengineering effort, we tailored 
them for this specific study. We collected survey data using the web surveys, actigraphy 
using the Actigraph watches, logging data using the event logger on the PDA and 
performance data using the ARES software, also on the PDA. By conducting an actual 
field study, we were able to experience firsthand the pressures facing a research team in a 
field environment. Through our own experience and observations, we were then able to 
modify the templates we had designed in order to provide maximum functionality to 
future research teams. The results of this OIS study serve as a validation of the 
reengineered efforts reflected in the “TO-BE” process. 

B. PHASE 1: COLLECT DATA SUB-PROCESS 

In the weeks preceding the OIS study, we completed tasks similar to those 
required by fatigue research students under the sub-process activities: Train Research 
Team and Prepare for Data Collection. Using the “TO-BE” process additions, we revised 
the Study Repository Template database on the HSIE web server 
(http ://www .nps .navy .mil/HumanEactors) and saved the newly revised OIS Study 
Repository database to the server. We entered the research team information and the 
study information using the input forms in the database. Once the decisions had been 
finalized on what data we would collect, we were able to modify the Institutional Review 
Board (IRB) proposal documentation template and submit the proposal for approval. We 
saved a copy of the IRB proposal documentation to the server. 

After the IRB proposal was approved, we modified the survey and log templates 
to fit the OIS Study. The Demographic questionnaire and Eark and Owl surveys were 
posted to the website and PARTICIPANT and EARK AND OWL tables in the OIS Study 

Repository database were set to be updated as data was entered into the web surveys. 
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The use of append table queries allowed the OIS Study database to automatically check 
for new updates on the web server and append them upon opening of the database. The 
PDA Event Log Template was modified to allow participants to log when they took the 
Actigraph watch off and put it back on, when they went down for sleep and when they 
awoke, and when they stood watch. The newly modified log was called Sleep_Log_OIS. 

Under Prepare Participant Activity, coordination was made with the OIS cadre to 
identify 20 participants for the study and to ask them to complete the web surveys. As 
they completed these, we could monitor their submitted survey data using the OIS Study 
Repository database. If any survey data field was incomplete, we were able to coordinate 
through the cadre to rectify the problem. As a result, we were able to collect all 20 of the 
participants’ survey data with minimal missing values. Participants completed online 
documentation the week prior to the study start date. Had we been actual fatigue 
researchers, the Lark and Owl data could also have been evaluated allowing the 
opportunity to connect momingnes s/eveningness information with each participant. As 
participants were added, the database automatically assigned participant ID numbers to 
each person. Using these numbers, we were able to conduct the tasks under the Prepare 
Equipment Activity. We initialized the Actigraph watches and assigned each to a 
participant using the database input form. We also prepared the ARES and logging PDA 
software for download to the PDAs. Because we were going to use the PDAs that had 
been issued to the participants by OIS, we could not prepare the PDAs until we arrived 
on site. 

Arriving a day before the actual Actigraph data collection was scheduled to begin, 

we were able to meet with participants to collect their PDAs and coordinate for 

redistribution of the devices. We also were able to clarify any missing entries made to 

their survey data which resulted in “already-cleaned” Demographic questionnaire and 

Lark and Owl data. We synchronized each participant’s PDA with toe standard Palm 

Desktop software that accompanies the PDA software. Since these PDAs were already in 

use by the participants, we could not rename them to conform to our naming convention 

for PDAs of HSILl, HSIL2, etc. which we had used in our template design. This would 

have required us to delete everything on their PDAs. As each PDA “user” name was 

added to our synchronizing software, we recorded this name into the PDA table of our 
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OIS Repository and added the corresponding Participant ID for each PDA. Once the 
PDAs were connected with Participant IDs, we loaded the ARES and logging software to 
each PDA. We also loaded the newly modified OIS Sleep Log file, 
Sleep_Log_OIS.PDB, and the OIS Sleep Log Input Lorm file, 
Sleep_Log_OIS_HDL.PDB, to each PDA. In preparing each log, we selected the 
appropriate participant ID number for each PDA using the HanDBase 3.0 desktop 
software and these two files. Although this participant’s “ID#” field displays on the PDA 
log view (Ligure 39), the user cannot modify it. 



Figure 39. PDA Quick Link Buttons 


When the data are collected, this participant ID number allows the researcher to 
identify whose data was downloaded. Similarly, using the registration menus on the 
ARES software on the PDA itself, we registered each participant using the corresponding 
participant ID and we set each to use the Commander Battery of tests. This battery 
includes a sleepiness scale, a reaction time test and a running memory test. Because the 
participants were in a classroom environment, we chose not to set their PDA alarms to 
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alert them to take their ARES tests though this technique proved to be helpful when we 
tested it at the HSIL. To assist participants, we did set the Memo Pad and To Do List 
buttons on the bottom of the PDA (Figure 39) to open the Log and ARES respectively for 
quick and easy data entry. 

The following morning, we distributed the Actigraph watches, PDAs and 
informed consent forms and instructed the participants on to the use of the PDA software. 
Once the signed consent forms were collected, we began the 5-day data collection phase. 
During the course of data collection, we periodically verified data integrity by collecting 
the Actigraphs and PDAs and downloading the data. One Actigraph device failed during 
the study and a replacement watch was given to the participant. Throughout this process, 
the participant continued to log his activities. 

Once the data collection period had ended, all Actigraphs and PDAs were 
collected. We synchronized the PDAs using the Palm Desktop software to collect the log 
and ARES data, deleted the ARES and logging software from the PDAs and then 
returned the PDAs to the participants. As we did this, we used the Collected Data and 
Equipment Form on the OIS Study Repository to track our progress. 

At the end of the Prepare for and Collect Data Sub-Process, we had PDA data 
files and Actigraph data files stored on the computer for each participant, signed consent 
forms and the Actigraph watches with stored data. Additionally, the OIS Study 
Repository database included completed and cleaned STUDY, RESEARCH TEAM, 
PARTICIPANT, and LARK AND OWL tables and cross-references for PDAs and 
Actigraphs to the participants in the PDA and ACTIGRAPH tables. Formatted reports 
were also available from the OIS Study Repository to assist in the data analysis phase. 

C. PHASE 2: PREPARE DATA FOR ANALYSIS SUB -PROCESS 

We began this phase with the Download PDA Data Activity. As a result of the 
PDA HotSync, each participant’s sleep log data included in the file Sleep_Log_OIS.PDB 
from the PDA was copied to the C^Palm\[PDA_name]\Backup location on the laptop 
hard drive. From the HanDBase desktop software, we opened each of these files and 
appended each participant to one consolidated Sleep_Log_OIS.PDB file. We then 
exported that data file to MS Excel and copied it directly to the LOG table in the 
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repository database. The pre-formatted Log Report in the database generated a 
consolidated, color coded log of each of the participant’s activities. Similarly, we 
prepared the ARES data by loading the ARSubjDB.PDB file of each participant stored in 
the C:\Palm\[PDA_name]\Backup location on the laptop. Unlike HanDBase however, 
the Data Manager software does not have an append function. Therefore, we copied each 
participant’s ARES data directly from Data Manager to the ARES DATA table in the 
OIS Study Repository database using the cut and paste function. We saved all of the log 
and ARES original .PDB files to the HSIE web server in the OIS Study Repository 
folder. 

Using the Actigraph interface device, we downloaded the actigraphy data using 
Act Millennium software and saved each file with the serial number as the filename as a 
.DAT file and .AMI file (e.g., SN01.DAT and SNOl.AMI). Because the Actigraph data 
files can only be opened from inside the Act Millennium and Action W software, there 
was no reasonable way to store this data into the database. Therefore, we stored both of 
these files on the web server. The ACTIGRAPH table in the database provides the key to 
identifying the corresponding Actigraph serial number(s) for each of the participants. 

Actigraphy data were trimmed and marked according to the Eog Report of the 

participants’ activity using Action W software. The data files were saved as .AMI files 

and .EBE files with an “A” added to the filename (e.g., SNOIA.AMI and SNOIA.EBE) to 

allow both the raw and trimmed data to be easily identified. All of these files were stored 

on the web server. Using the summary statistics of Action W for each participant, the 

average number of hours of sleep per night calculated for the participants was recorded. 

This average of six hours was used to precondition the preceding three days of the 

template schedule using the EAST software. Each .EBE file was then loaded into the 

template EAST schedule with the preconditioned preceding three days. Each 

participant’s EAST data was saved as a EAST schedule using the .EAS file extension 

(e.g., SNOl.EAS). These EAST schedules were also added to the OIS Study Repository 

folder on the web server. Eor the participant whose watch had failed and had been issued 

a second watch, we were unable to combine the data from both watches into one Action 

W data file; both sets of data were cleaned and stored into two respective files based on 

the log. Because the participant kept an accurate log including the time between the first 
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watch failing and the second watch beginning to record data, it was possible to manually 
input the sleep/wake from the first watch’s failing through the end of the study period 
into FAST using the Action W file from the second watch and the log. This provided an 
accurate measure of the participant’s predicted effectiveness and made it possible for 
keep this participant’s data in the study. After all Actigraph data were “cleaned” and 
predicted cognitive effectiveness was modeled in FAST for each participant, screen 
captures of the trimmed Action W files and the FAST schedules for each participant were 
also imported into a PowerPoint presentation and saved on the server. 

By the end of the Prepare Data for Analysis Sub-Process, all participants’ raw and 
“cleaned” data files were well-organized on the web server for easy access as was a 
summary presentation file including the actigraphy and FAST performance effectiveness 
for each participant. The OIS Study Repository database included updated LOG and 
ARES tables and the data from all 20 of the participants was prepared for analysis. As 
projected by our KVA calculations, we expected the cycle time for this sub-process to 
decrease by half. Because of the standardized templates and storage methods as well as 
the efficiently coded log data easing the burden of “cleaning” actigraphy, the actual cycle 
time for this sub-process for this study was less than one forth of the corresponding sub- 
process cycle time in the “AS-IS” model. 

D. PHASE 3: ANALYZE AND PUBLISH SUB-PROCESS 

Because the data analysis portion of this sub-process did not change from the 
“AS-IS” process to the “TO-BE” process, we did not conduct an in-depth analysis of the 
sleep data. We did, however, request the assistance of research analysts to verify that the 
data could be analyzed efficiently in their stored formats. One of the researchers that 
assisted us in this evaluation was ET John Nguyen who had conducted the USS JOHN C. 
STENNIS Study. In his assessment, the prepared data were ready for analysis. As a 
result of the standardized formatting and efficient activity logs, he estimated that he could 
have saved months of time had he had this “TO-BE” process available to him prior to 
conducting his study. A second researcher looked at the ARES data and determined that 
from its stored format in the repository database, it could have easily been evaluated to 
determine possible correlations between performance and fatigue. There were some 
participants that did not complete the battery of tests the recommended three times per 
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day. Because we were more interested in whether the data could be collected and 
downloaded efficiently, we did not adequately emphasize the importance of completing 
all required testing. In the future, researchers can download ARES data daily to ensure 
participant compliance for complete analysis. 

Due to the ease of “cleaning” the Actigraph data, we were able provide the 
command at OIS some preliminary feedback on the participants’ sleep/wake activity as 
well as performance effectiveness during the collection period. This was the first time 
such immediate feedback could be provided to a sponsoring organization. As a result of 
this feedback, the command requested the Actigraph data be collected for an additional 
period to evaluate the participants’ performance during a battle drill exercise. Though no 
formal data analysis feedback was requested, the command was able to see the potential 
value provided by this fatigue study process to maximize student performance given their 
limited time in training. Another study may be warranted to answer specific questions 
for the OIS command in regard to fatigue management. Data collected from this study 
could be used as a baseline for comparison since it is available in the repository. 

E. LESSONS LEARNED 

As a result of this validation study, we were able to experience, firsthand, the 
challenges of conducting research on human in the field. In this context, we have made 
some valuable observations that impact future studies using our reengineered process. 
The following paragraphs summarize these observations in four major areas: 1) 
advantages of the PDA event logger, 2) PDA issues, 3) Actigraph issues, and 4) ARES 
issues. 

1. Advantages of Event Logger 

Trimming raw Actigraph data is time consuming using Action W software. To 
accurately represent a participant’s data, researchers compare the data recorded in the 
sleep and activity log to the Actigraph data to verify whether the data are sleep or wake 
epochs and to record when an individual was down for sleep. When using a paper log, 
the researcher has to either transpose the log data to a spreadsheet or use the paper itself 
to compare with the Actigraph. In both methods, extracting the useful information from 
the log is tedious. In cases such as the USS JOHN C. STENNIS study, the inaccuracy of 
the paper bgs resulted in the researcher having to eliminate the use of logs from his 
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“cleaning” process only after many hours were spent trying to glean useful information 
from them. When they finally were eliminated, the researcher had to use a technique for 
trimming sleep data based on assumptions. Using this standard method for trimming data 
in the Action W software, researchers coded epochs where activity levels fell below 50 
per epoch for 20 or more minutes as sleep periods. If the activity registered as a 0 for 10 
minutes or more, the researcher assumed the participant had removed the Actigraph 
watch and coded the data as sleep or wake periods based on the duration and time of day. 
All other activity periods were classified as wake. 

Using the PDA log, participants were much more likely to log their daily events 
because the method was easy for them to do so. When comparing log data with 
Actigraph data, the Log Report made “cleaning” the raw Actigraph data very simple. 
Figure 40 shows a screen capture of the method used during this “cleaning” process. The 
top window is the participant’s raw Actigraph data in Action W software and the bottom 
window is his corresponding log data from the Log Report from the OIS Study 
Repository database. The researcher looks for the highlighted events and records them 
using the Action W tools. Using the participant number 52’s data from the OIS study, we 
were able to “clean” his data in less than 5 minutes using this method. Using the standard 
trimming technique that the USS JOHN C. STENNIS researcher used to “clean” the data 
without a log on this same participant, it took approximately 15 minutes. These results 
were even a greater improvement than we had estimated in our KVA analysis 
(APPENDIX A). We tested these results on all of our participant data and found that on 
average, the standard trimming technique took more than three times as long to “clean” 
Actigraph data than when using the PDA log. 
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Figure 40. “Cleaning” Actigraph Data Using Action W and PDA Log Report 


Not only is the PDA log method faster than the standard coding technique but it 
also appears significantly more accurate in some cases. Because of the convenience of 
the PDA log, participants are more likely to maintain their logs. Additionally, because 
the researchers can quickly compare log data with Actigraph data, they can routinely 
conduct data downloads to assess the quality of data collection and take corrective 
measures if needed. If there are unexplained data points, the researchers can clarify the 
issues with the participants while still there and while events are still fresh in their 
memory allowing the logs to more accurately reflect their behavior. 

To emphasize the impact the log has on analyzing a participant’s data, we 
compared the standard method of coding used in prior studies to the method of using the 
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PDA log. Using participant 52 from the OIS study again, Figure 41 shows this dataset 
trimmed with the standard coding technique. The red underlined epochs represent sleep 
periods; all other times are coded as wake. The aqua filled box periods represent the 
times a participant was down for sleep. Using this same participant in a comparative 
example, the Actigraph data were trimmed using the PDA based log (Figure 42). 
Because the activity levels did appear unusually high for sleep epochs, we asked 
participant 52 to verify the sleep log and verify our assessment of his actigraphy while 
still on site. He confirmed the events he had logged and assured us he had not awoken 
during his nights of sleep. Figure 43 shows a side by side comparison of the Actigraph 
data trimmed using both techniques. Without the log, many epochs were coded as wake 
as they did not fall below the threshold of 50 for at least 20 minute intervals. In the data 
“cleaned” using the standard coding technique, it appears as if the participant is getting 
much less sleep than he actually was. 



Tue 4/U2003 




Thu 4/3/2003 




Figure 41. Actigraph Data Trimmed Using Standard Method 
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Trimmed Using Standard Method Trimmed with PDA Log 

Figure 43. Side By Side Comparison of Actigraph Data 


We imported the data from both methods into FAST sehedules to model 
participant 52’s predicted effectiveness. Figure 44 shows the FAST model of the 
Actigraph trimmed using the standard method and Figure 45 shows the FAST model 
using the PDA log to trim the data. The red line indicates the sleep reservoir as it is 
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depleted and refilled according to the sleep/wake epochs. Figure 46 shows a side by side 
comparison of the FAST models. The modeled effectiveness of the two is decisively 
different. The additional sleep coded using the PDA log and the testament of the 
participant immediately after the data had been collected to verify accuracy accounts for 
a much higher predicted effectiveness level. Without any log, the model indicates 
predicted effectiveness far below acceptable levels. Though it appeared from his raw 
Actigraph data that participant 52 did not get much rest, according to his report, he did. 
This report made by the participant is his subjective account. However, the assumptions 
made by the researcher without reference to a log are also subjective and arguably less 
accurate. Subsequent examination of the Actigraph used by participant 52 showed that 
the device needed factory calibration. 



Figure 44. FAST Model without a Log 
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Figure 45. FAST Model with a PDA Log 



Trimmed Using Standard Method 


Trimmed with PDA Log 


Figure 46. Side By Side Comparison of FAST Models 


Because our focus during the validation study was on collecting and storing data 
efficiently, we did not concentrate on the actual fatigue data as sleep researchers would 
have. Had we been interested in answering fatigue related questions, we could have 
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conducted routine data downloads of Aetigraph and log data and eondueted quiek 
analysis to ensure participants are logging events effeetively. Using the old method, the 
time it took researehers to verify Aetigraph data with the log data made this eomparison 
impossible to do while on site without interrupting the study. The addition of the PDA 
log allows researehers to eonduet this data verifieation between Aetigraph data and 
logged events very easily. Any issues noted by the researeh team ean be resolved 
immediately with the partieipant thus inereasing the aeeuraey of the analysis. 

2. PDA Issues 

As part of our reengineering researeh, we evaluated various PDA handhelds and 
ehose the Handspring TREO for our reengineered proeess. The laptops we used were 
equipped with Universal Serial Bus (USB) ports. When eomparing ease of using the 
HotSync funetion, the Handspring versus the Palm products we evaluated seemed easier 
to use. Because the HanDBase and ARES software work with any PDA using the Palm 
OS, however, the proeess was not dependent on the PDA model. When we ehose OIS as 
our validation study loeation, we understood that students and eadre were issued Palm 
PDAs upon arrival. Beeause we had not yet bought PDAs for HSIE, we deeided to use 
the partieipants’ Palms. We requested from OIS a sample Palm PDA so we eould load 
and test our software to ensure eompatibility. A Palm ml00 arrived and although the 
display was blaek and white, all software seemed eompatible. The Palm mlOO eradle 
used a serial eonneetor to the laptop for HatSyne funetions. When we arrived on site, we 
found that all partieipants were using the Palm V version of PDAs. When we tried to 
perform a HotSyne funetion with our laptop using the Universal Serial Bus (USB) 
connection from the Palm V eradle to the laptop and Palm Desktop software, there were 
errors that eould only be correeted by uninstalling and reinstalling the Palm software. 
Even after this, we experieneed occasional HotSync errors with the Palm V using the 
USB port. Additionally, we had to spend hours eonfiguring the Palm V’s with the data 
colleetion software the evening before eolleetion began. To avoid eorruption of the 
partieipants’ saved information on their PDAs, we ehose to use their PDA names they 
had established and deviate from our proeess’ naming eonvention (e.g., HSIEl, HSIE2, 
ete.). This eaused us to have to ehange the repository database PDA STATUS table and 
populate it with the OIS PDA names. Beeause many partieipants used their last name as 
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their PDA name, this collected data became sensitive since the HSIL policy prohibits 
storing any operational data with information identifying the participants other than their 
participant ID number. In order to store the data on the HSIL server, we had to delete 
these PDA names. 

We recommend for future studies that the HSIL purchase 30 of the Handspring 
TREO models. When installing each PDA on the HSIL operational study laptop we 
recommend that PDAs be named according to our naming convention: HSILl through 
HSIL30. Each should be loaded with the HanDBase and ARES software. Additionally, 
we recommend loading the Sleep Eog Template and its accompanying input form created 
with HanDBase software onto each PDA. Eor both the log and the ARES programs, the 
participant ID numbers should be added. These ID numbers should correspond to the 
PDA number. Eor example, on the PDA named HSIEl, the ARES registered user should 
be 01 and the “ID#” field on the log form should be 01. This way, all data downloaded 
from ARES and the event log programs during the HotSync function will be 
appropriately marked with a participant ID number that corresponds to the PDA ID in the 
Study Repository database. 

In the event that participants are already issued PDAs from a U.S. governmental 
agency, we recommend that the research team attempt to accommodate the participants 
and use their issued PDAs so as not to further inconvenience the participants by requiring 
them to carry a second PDA during the data collection phase. In this case, it is essential 
that the research feam acquire a sample of the PDA model used by the participants to 
ensure compatibility with both software and hardware. Additionally, the research team 
should allow enough time after arrival on site and prior to data collection to configure the 
participants’ PDAs accordingly. Einally, the research team must ensure that the data 
saved from each PDA after each HotSync under the Palm directory on the laptop is 
erased and any identifying PDA ID names in the Study Repository database are erased as 
well to protect the participants’ identity. 
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Despite the technical obstacles, the use of PDAs undoubtedly improves the “AS- 
IS” process. Whether using the PDAs from the HSIL or participants’ PDAs, the ability to 
collect event log data and ARES data provides a tremendous advantage to the research 
team. 

3. Actigraph Issues 

Collecting actigraphy using the Actigraph and associated software posed unique 
challenges. Issues involving the Actigraph watches themselves as well as observations of 
other hardware and software became apparent to us during the study. 

The Actigraph watch itself was bulky for the participants as they noted on their 
exit surveys; some cadre, unaware of the study, required the student participants to 
remove their Actigraphs for inspections. This resulted in a few participants failing to put 
the watch back on for several hours. A smaller more watch-like Actigraph with long 
battery life and low failure rates would clearly be better suited for field studies. 

The interface box for the Actigraph requires a serial port connection and is not 
available for use with a USB connector. Because many laptops are no longer being made 
with serial ports, this may become an issue in the future. In a similar incompatibility 
issue, the software accompanying the watches from the manufacturer is a DOS based 
program that will not run on any Windows Operating System higher than Windows 95. 
The Act Millennium software offered by AMI, the marketing company of the Actigraph 
watches, as a Windows alternative is compatible with Windows XP. However, the 
initialization of and downloading of data from the watches are not intuitive processes. 
Researchers must be very familiar with the configuration settings of this software prior to 
collecting data. 

Once the raw data have been downloaded using the interface unit and Act 
Millennium, they are saved as a .DAT file and then saved as an .AMI file to allow Action 
W software to read the data. Once trimmed in Action W, the data file has to be saved as 
an .EBE file to be read with EAST. After the research team has built a schedule in EAST 
covering the collection period, the .EBE file is imported into the schedule. After it has 
been “cleaned” in EAST, the data are again saved with a .EAS extension so it can be 
opened as a schedule in the future. Because of the unique data formats for these 
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programs, the data could not be saved in the database for export. Instead, all versions of 
data with unique file extensions were saved in the same folder as the database on the web 
server. We tried to save the hyperlink to this folder location in the database but the files 
could not be associated with the programs to open automatically. Once a file is saved 
from Act Millennium as an .AMI file for Action W for example, we could not double 
click on that file and have Action W open and load the data. This process of opening 
programs, manipulating data, saving in new file formats to new locations on the server 
proved to be tedious and cumbersome. This inefficiency caused delays in preparing data 
for analysis that could easily be overcome by allowing import and export of universal 
data formats in all of the programs. 

To ensure data integrity, researchers must be familiar with Act Millennium, 
Action W, and FAST software programs well before the data collection phase begins. 
Even with a comprehensive understanding, the process for tracking raw and “cleaned” 
data files quickly becomes overwhelming to the inexperienced researcher. 

4. ARES Issues 

Use of the ARES test battery requires researchers to plan for a significant increase 
in PDA power usage, unforeseen PDA system failures and unique data storage 
requirements. Despite some difficulties encountered using ARES software, the data from 
the performance tests were collected and the use of ARES is now a viabb metric 
available for future studies. 

Taking the ARES tests battery drains the PDA power rather quickly. As a result, 
researchers must be sure to remind participants to charge their PDAs daily. If 
participants’ PDAs require disposable batteries, it is recommended that the research team 
maintain an adequate supply of these batteries on hand throughout the entire data 
collection period. 

As a result of a bug in the ARES software, two PDAs locked up during the OIS 
study. The data in one device was unaffected because the participant brought the PDA to 
us and we initiated a soft reset. However, the data in the second device was lost because 
the participant inadvertently initiated a hard reset. Eortunately, it was on the first day of 
collection so only data one set of ARES data was lost. Eor future studies, researchers 
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must obtain the latest version of ARES software to minimize the risk of bugs and ensure 
participants are aware of hard and soft reset procedures for PDAs in the event there is a 
problem. It is also recommended that short procedure cards be inserted into the PDA 
cases provided for the study. This card can contain contact information for researchers, 
soft reset procedures, and brief “how-to” instructions for completing ARES tests and 
logging events. 

When downloading ARES data, there are two means available to the researcher. 
Eirst, the desktop Data Manager program can be opened while the PDA is connected to 
the computer and the database file can be directly loaded into Data Manager. The second 
method is by using the HotSync function. Because the log data also have to be 
downloaded from the PDA, the HotSync method is more convenient. Once the file is 
downloaded as C\Palm\[PDA_Name]\ARSubjDB.PDB, the Data Manager program has 
to be opened and only from there can the researcher load this data file. The Data 
Manager Program will not accept any other .PDB file than one named ARSubjDB.PDB. 
Therefore, all of the ARES raw data files from the participants cannot be stored in the 
same folder. Additionally, because the files all have the same name, there is 
understandable confusion manipulating these files and copying and storing the data to the 
database. Additionally, once the raw data are consolidated in the ARES table in the 
database, it can no longer be loaded into Data Manager but only used in programs like 
MS Excel. This significantly hinders the research team by requiring them to track many 
storage locations for the same data. 

Though our data has yet to be analyzed thoroughly, we are confident that 
collecting ARES data on the PDA is an option now available to HSIE researchers. 
Diligent monitoring of and acquiring the new releases of upgraded versions of ARES 
software is imperative as some of these obstacles may be resolved in later software 
versions. 

F. CONCLUSION 

The OIS Study served to validate the improvements made in the “TO-BE” 

process. The data collection techniques do strea ml ine the process, making the data 

analysis more efficient and accurate. While some recommendations for particular 

hardware and software used must be considered, the results of this study show that 
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progression from the “AS-IS” to the “TO-BE” process is not only feasible but necessary. 
Using the reengineered system will allow more studies to be conducted annually with 
more valuable analysis due to better accuracy. Additionally, the storage repository 
allows a significant capability of conducting meta-analyses whose value has yet to be 
realized. The importance of fatigue research to the military, in particular, demands this 
immediate change from the current process to the reengineered one. 
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m CONCLUSIONS 


A. SUMMARY DISCUSSION 

The tremendous challenges faced by scientists conducting human research are 
further magnified when research is moved from the controlled laboratory to the field 
environment required by operational research. In the particular field of human 
performance and fatigue research, the unique intricacies of data collection represent some 
of the most complex in the field of human research. Serving as a Proof of Concept for a 
representative human research process, the process of conducting human performance 
and fatigue research at the HSIL at NPS was used to substantiate effective data collection 
and storage techniques. Applying current information technologies, we were able to 
demonstrate a significant improvement in productivity from the current “AS-IS” process 
to the reengineered “TO-BE” process. The methods used in this Proof of Concept could 
be applied to many data collection and storage processes in the broader field of human 
research thus refining the effectiveness of operational studies as researchers as they strive 
to improve human performance. 

Using a three-tiered approach in the redesign, various graphical user interface 
means were utilized to collect and store data in a relational database housed on a NPS 
web server. Replacing paper surveys, web surveys and PDA-based input forms were 
incorporated to collect participant data. Similarly, database forms were designed to allow 
researchers to record details of a particular study. A PDA-based version of human 
performance software was used to collect performance data while wrist activity monitors 
containing accelerometers recorded participant activity levels. The database served not 
only as a storage facility to organize and maintain the collected data but also provided the 
research team with a standardized method to track their entire research process. As a 
result of the insertion of various information technologies, the reengineered process 
included automating current sub-processes and tasks and more importantly enabling new 
processes to be incorporated. With the conservatively estimated ROK improvement from 
12% realized by the current process to 26% offered by the reengineered process, 
researchers at the HSIL at NPS will be able to conduct more studies per year while each 


89 



study produces more accurate data analysis results. Additionally, the standardized 
storage means provided in the “TO-BE” process enables researchers to conduct meta¬ 
analysis; a capability not possible with the “AS-IS” process. 

B. RECOMMENDATIONS 

Because reengineering is, in itself, an iterative process, the prototype system 
developed for this Proof of Concept was the first step. Near-term initiatives taken at the 
HSIL at NPS to implement this redesign will result in greater overall improvement to the 
process. Efforts made to increase compatibility between systems by human performance 
and fatigue research hardware and software developers will further improve not only the 
process at NPS but the more extensive field of human performance and fatigue research. 
Einally, to transform the impact human research has on improving human performance 
under constraining conditions, system developers can take steps to design interoperable 
and unobtrusive collection means that allow for immediate analysis of human systems 
thus providing decision makers with the feedback required to optimize human resources. 

1. Near-Term Initiatives to Improve the Process at the HSIL at NPS 

In order to capitalize on the reengineering effort, it is necessary for the HSIE to 
employ an IT professional to implement the prototype system described in the “TO-BE” 
process and provide adequate system maintenance. This thesis serves as a guide for the 
research team, including the additional IT professional, at the HSIE to package a “How- 
To” type toolkit for future researchers to utilize when conducting field studies. The range 
of techniques for data collection extend beyond the methods used in this Proof of 
Concept system and each should be modeled to provide researchers an inventory of types 
of data to collect and the instructions and means to collect each in various operational 
environments. Secondly, templates should be created for each additional data collection 
method available to the HSIE researchers but not used in this Proof of Concept. Einally, 
a web-based feedback form should be created to capture the techniques used by research 
teams including both successful and unsuccessful trials. This cumulative feedback 
repository will include the data and analysis of all studies conducted at NPS and should 
capture all lessons learned by each research team thus creating a venue for continual 
improvement of human performance and fatigue research processes at NPS. This transfer 


90 



of corporate knowledge to IT resources will provide future research teams an invaluable 
source of information allowing them to maximize the potential effects of their research 
on human performance. 

2. Immediate Improvements to Human Performance and Fatigue 
Research Hardware and Software 

Though current hardware and software developments in the field of human 
performance and fatigue research have only recently made operational studies possible, 
straightforward improvements to the existing hardware and software could have a 
tremendous impact on the productivity of the human performance and fatigue research. 
Storing data from various data collection media into compatible formats will drastically 
reduce the time and effort required of researchers to prepare their data for analysis. 
Further, an interface application to provide one point from which the researcher can 
access all software programs would ease the burdens associated with data “cleaning” thus 
allowing more time to spend on analyzing the collected data. This application should 
include the tools required to download, “clean,” and analyze data as well as the means to 
efficiently both store the data and published the analysis. A more far reaching, yet 
attainable, goal should be to create a single suite of hardware and software that allows 
researchers to choose from an inventory of unobtrusive data collection media required for 
their research and easily tailor that media to fit the operational constraints of their specific 
data collection effort. This data collection means should incorporate wireless 
technological advances to allow data transfer without human intervention thus further 
increasing accuracy and saving time. The suite should also provide researchers with a 
variety of analysis tools to enable more comprehensive thus more revealing analysis. As 
new hardware monitoring devices are designed for human performance and fatigue 
research, they must be developed with system interoperability in mind. 

3. Long Term Designs for Human Research Product Developers 

To reach beyond data collection and analysis by offering solutions to mitigate 

obstacles, developers in the field of human research can make a tremendous impact 

toward improving the human condition. With the continual development of biomedical 

monitoring devices, developers can design products that not only monitor various human 

systems but offer self-analyzing feedback of the particular system’s performance. From 

these various human system sensors as well as environmental monitoring media, 
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developers will be able to create systems to evaluate the feedback and provide 
recommendations for physical or behavioral modifications to improve human 
performance. As a result of their complexity, humans provide invaluable and often 
irreplaceable additions to many processes. Optimizing human performance under 
constrained conditions is the goal of industry leaders and military commanders alike. 
Focusing human research product development on this goal, researchers will have the 
tools required te) offer leaders options for resolving conflicts that inhibit optimal human 
performance. 
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APPENDIX A. KVA “AS-IS” ROK 
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Process: NPS Executing Fatigue Studies 


Sub Processes 

Weighted 

Learning 

Time 

Total 

Cost 

As-Is 

ROK 

To-Be 

ROK 

Dramatic 

ROK 

Collect Data 

289 

3300 

9% 




1810 

4246 


43% 



395760 

370425 



107% 

Prepare for Analysis 

1158 

8278 

14% 




3075 

9185 


33% 



1204500 

2062500 



58% 

Analyze and Publish 

1161 

10192 

11% 




6885 

32082 


21% 



254250 

1122413 



23% 







TOTAL 

2608 

21770 

12% 




11770 

45513 


26% 



1854510 

3555338 



52% 
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APPENDIX B. WEB SURVEY PROTOTYPES 


Lark and Owl Web Survey 


Naval Postgraduate School Bll^= 

H 0 n t € r “ V . C a i • c r n a 


Lark and Owl 

Instructions: 

Take the following survey to help us determine if you are a lark, and owl or somewhere in 
between. These questions should be answered as if you were at home and not deployed? 

Last Name I Type your last name 

Circadian Identity: 

1. During your last vacation week, how often did you get up iater than pianned or have 
difficuity getting ready on time even though you went to bed at your regular time? 

^ It never happened 

^ It happened once 

^ It happened two or more times 

^ It happened more than three times 

2. When you have no commitments the next day, at what time do you go to bed compared 
with your usual time? 

^ Seldom or never later 

^ Not more than one hour later 

^ Between one and two hours later 



Demographic Web Survey Template 


Naval Postgraduate School 

M 0 n t e r s V . C a i •' c r n a 


Local Demographic Survey 
Instructions: 

Please fill out all blocks in this survey, when finished please press 
the "SUBMIT" button 

1. NAME: Last; | First: | MI:| 

2. Rank: fo^ 

3. Service: fuSN 3 Prior Service I” Yes 

4. E-mail Address: |N/A 

5. Home Phone: | 

(SSS)S5S-5555 

6. Pager Number/Cell; |n/a 

(555)555-5555 

7. Home Address:! 

City:| State: ICA “^1 

8. Age: j 

9. Gender: ^ Male ^ Female 

10. Height: M ^IfeetlO "clinches (round to the nearest inch 

11. Weight:! lbs 

12. Do you wake up to an alarm clock? yes No 

13. What time of day do you typically leave campus? |1200-1^00 ^ 

14. In the past four weeks, how much have you been able to workout? (physical activity) 

^ 0-1 times per week 4-5 times per week 
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Morningness/Eveningness Scale 


Directions: This scale is designed to help you identify a person’s tendency toward a 
morning (lark) or evening (owl) circadian rhythm pattern. 

1. During your last vacation week, how often did you get up later than planned or 
have difficulty getting ready on tie even though you went to bed at our regular time? 

A. It never happened 

B. It happened once 

C. It happened two or more times 

D. It happened more than three times 

2. When you have no commitments the next day, at what time do you go to bed 
compared with your usual time? 

A. Seldom or never later 

B. Not more than one hour later 

C. Between one and two hours later 

D. More than two hours later 

3. Suppose that you have decided to exercise twice a week with a friend. The only 
time your friend can make it is from 7:00 a.m. to 8:00 a.m., twice a week. Assume you 
decide to go at these times. Taking into account how you usually feel in the morning, 
how would you do? 


4. 

sleep? 


A. Very well 

B. Well 

C. Poorly 

D. Very Poorly 

At what time in the evening do you usually start feeling tired and in need of 

A. 8:00 p.m. - 9:30 p.m. 

B. 9:31 p.m. - 10:45 p.m. 

C. 10:46 p.m. -12:30 a.m. 

D. 12:31 a.m. -1:45 a.m. 

E. 1:46 a.m. - 3:00 a.m. 


5. Suppose that you were able to choose your own working hours. Assume that your 
job was interesting and paid according to results. Which one of the following 3-hour 
blocks would be your most preferred work time? 


A. 

B. 


4:00 a.m. - 7:00 a.m. 
7:00 a.m. - 10:00 a.m. 
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C. 11:00 a.m. - 2:00 p.m. 

D. 4:00 p.m.-7:00 p.m. 

E. 9:00 p.m. - 12:00 Midnight. 

6. One sometimes hears about “feeling best in the morning” or “feeling best in the 
evening” types of people. Which do you consider yourself? 

A. Definitely a “morning” type 

B. More a “morning” than an “evening” type 

C. More an “evening” than a “morning” type 

D. Definitely an “evening” type 

SCORE GUIDE 


Question 

Answer 

Score 

Question 

Answer 

Score 

No. 



No 



1 

A 

6 

4 

A 

7 


B 

5 


B 

6 


C 

5 


C 

5 


D 

4 


D 

4 





E 

3 

2 

A 

7 

5 

A 

7 


B 

6 


B 

6 


C 

4 


C 

5 


D 

3 


D 

4 





E 

3 

3 

A 

7 

6 

A 

9 


B 

6 


B 

7 


C 

4 


C 

3 


D 

3 


D 

1 


SCORE SCALE 

17 to 19 Extreme Owl 

20 to 23 Moderate Owl 

24 to 29 Mild Owl 

30 Neither Owl nor Eark 

31 to 36 Mild Eark 

37 to 40 Moderate Eark 

41 to 43 Extreme Eark 
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APPENDIX C. STUDY REPOSITORY TEMPLATE DATABASE 

RELATIONSHIPS 


Actigraph StatusActigraph 


Actigraph 

Status 

Actigraph 

Actigraph ID 1 

? actigraphID 

Attributes: 

Enforced 

RelationshipType: 

One-To-Many 


ParticipantActigraph 


Participant 

Actigraph 

ParticipantID 1 

? participantID 


Attributes: Enforced 

RelationshipType: One-To-Many 


ParticipantARES Data 


Participant 

ARES Data 

ParticipantID I 

? participantID 


Attributes: Enforced 

RelationshipType: One-To-Many 


ParticipantCollected Equipment and Data 


Participant 

Collected 


Data and 


Equipment 

ParticipantID I 

1 ParticipantID 

Attributes: 

Unique, Enforced 

RelationshipType: 

One-To-One 
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ParticipantLark and Owl 


Participant 

Lark and 
Owl 

ParticipantID 1 

? ParticipantID 

Attributes: 

Enforced 

RelationshipType: 

One-To-Many 


ParticipantLog 


Participant 

Log 

ParticipantID I 

? ParticipantID 


Attributes: Enforced 

RelationshipType: One-To-Many 


ParticipantPDA 


Participant 

PDA 

ParticipantID I 

? participantID 


Attributes: Enforced 

RelationshipType: One-To-Many 


PDA StatusPDA 


PDA Status 


PDA 

PDA ID 

I 

? PDA ID 


Attributes: Enforced 

RelationshipType: One-To-Many 
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Research TeamResearch_Team_Study 


Research 

Research_Team_Study 

Team 


ResearcherlD 1 

? ResearcherlD 


Attributes: Enforced 

RelationshipType: One-To-Many 


StudyParticipant 


Study 


Participant 

Study ID 

1 

? Study ID 


Attributes: Enforced 

RelationshipType: One-To-Many 

Study Research_T eam_Study 


Study 


Research_Team_Study 

StudylD 

I 

? StudylD 


Attributes: Enforced 

RelationshipType: One-To-Many 


107 















THIS PAGE INTENTIONALLY LEET BLANK 


108 



APPENDIX D. PROTOTYPE EORMS AND REPORTS EROM 
STUDY REPOSITORY TEMPLATE DATABASE 


Main Menu Form 


® Main Menu Form : Form 



Update Actigraph Status 


Update PDA Status 


Input Study Info 


Input Research Team Info 


Demographic Survey 
Lark or Owl Survey 
Assign Actigraphs 
Assign PDAs 
Collect Equipment 





Participant Information 


Assigned Actigraphs Report 


Assigned PDA Report 


Actigraph Sign-Out 


Log Report 


iExit Databiasei 


Actigraph Status Input Form 


S Actigraph Status Input Form 



► 


Actigraph ID 


NAVAL 

POSTGRADUATE 

SCHOOL 


SNOl 


Date Battery Changed |03/27/03 
(mm/dd/yy) 


Timestamp | 3/28/2003 11:34:14 AM 

Record; H I H i T ► I n |»*| of 25 
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PDA Status Input Form 


G1 PDA Status Input Form \ 


^Xj 



NAVAL 

POSTGRADUATE 

SCHOOL 


PDA ID 
ARES Loaded 


HSILl 


[7 


HanDBase Loaded 


Log Database Loaded ^ 

Record; i< I ^ 1 1 T ► I »i !►*! of 19 


Study Input Form 



no 










































Research Team Input Form 



Participant/Demographic Input Form 



III 





























































Lark or Owl Input Form 


M Lark or Owl 




naval 

POSTGRADUATE 
SCHOOL 


Participant's Last Name; 
Participant’s ID: 


U 


1. During your last vacation week, how often did you get up laterthan planned or have difficulty getting ready on 
tie even though you went to bed at our regular time? 

It never happened 
I It happened i-3timesi 
It happened more than three times 


When you have no commitments the next day, at what time do you go to bed compared with your usual time? 


C' Seldom or never later 
^ Not more than one hour later 
(* Between one and two hours later 
More than two hours later 


3. Suppose that you have decided to exercise twice a week with a friend. The only time your friend can make 
it is from 7:00 a.m.to 8:00 a.m., twice a week. Assume you decide to go at these times. Taking into account 
how you usually feel in the morning, how would you do? 


Record: H I < I F 


^ Very well 
(* Well 
Poorly 

1 ► I ►!\>*\ of 23 




Assign Actigraph Input Form 


S Assign Actigraph Input Form 


NAVAL 

POSTGRADUATE 

SCHOOL 



l asMi -I 


Actigraph Number 
Participant's Last Name |Smith 
Participant's ID 


3 


Record; l< I < I P 


ID 

rr 

— 

Previous | 

Next 1 

Close 1 

zl 

1 t 1 H \>*\ 

of 23 
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Assign PDA Input Form 



Collected Equipment and Data Checklist Form 


El Collected Equipment and Data Checklist 



NAVAL 

POSTGRADUATE 

SCHOOL 




► 

Lastname 

ISJiilHi 

J 



ParticipantID 

1 1 




Consent Forms Signed 

[7 




Collected Actigraph 

W 




Collected PDA 

R? 




Previous | Next | Close | 



Record; M | ■ |i T ► | ►! |t*| 

of 20 
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